Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 19:55:53.066726 17668 test_util.cc:268] Using random seed: -541169450
I20240501 19:55:53.072887 17668 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:55:53.072983 17668 ts_itest-base.cc:113] --------------
I20240501 19:55:53.073024 17668 ts_itest-base.cc:114] 5 tablet servers
I20240501 19:55:53.073055 17668 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:55:53.073067 17668 ts_itest-base.cc:116] --------------
2024-05-01T19:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:53Z Disabled control of system clock
I20240501 19:55:53.081274 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:46087
--webserver_interface=127.17.65.62
--webserver_port=0
--builtin_ntp_servers=127.17.65.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.65.62:46087 with env {}
W20240501 19:55:53.179857 17680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:53.180061 17680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:53.180141 17680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:53.183326 17680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:53.183409 17680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:53.183446 17680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:53.183502 17680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:53.187435 17680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.65.62:46087
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:46087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.65.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:53.188285 17680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:53.189365 17680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:53.196983 17685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.197602 17688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.198587 17686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:53.199396 17680 server_base.cc:1034] running on GCE node
I20240501 19:55:53.199862 17680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:53.200744 17680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:53.201887 17680 hybrid_clock.cc:648] HybridClock initialized: now 1714593353201876 us; error 23 us; skew 500 ppm
I20240501 19:55:53.208899 17680 webserver.cc:460] Webserver started at http://127.17.65.62:44431/ using document root <none> and password file <none>
I20240501 19:55:53.209422 17680 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:53.209477 17680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:53.209650 17680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:53.211243 17680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0a0381278c204add884696237c5baf99"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.211671 17680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0a0381278c204add884696237c5baf99"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.214664 17680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:55:53.216756 17694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.217634 17680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 19:55:53.217741 17680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0a0381278c204add884696237c5baf99"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.217813 17680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:53.238294 17680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:53.238945 17680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:53.239125 17680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:53.245434 17680 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.62:46087
I20240501 19:55:53.245440 17746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.62:46087 every 8 connection(s)
I20240501 19:55:53.246440 17680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:55:53.248862 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:53.253429 17747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99: Bootstrap starting.
I20240501 19:55:53.255499 17747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:53.256112 17747 log.cc:824] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:53.256572 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 17680
I20240501 19:55:53.256716 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:55:53.257753 17747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99: No bootstrap required, opened a new log
I20240501 19:55:53.260538 17747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0381278c204add884696237c5baf99" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 46087 } }
I20240501 19:55:53.260728 17747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:53.260784 17747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0381278c204add884696237c5baf99, State: Initialized, Role: FOLLOWER
I20240501 19:55:53.261158 17747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0381278c204add884696237c5baf99" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 46087 } }
I20240501 19:55:53.261266 17747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:53.261330 17747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:53.261426 17747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:53.262116 17747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0381278c204add884696237c5baf99" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 46087 } }
I20240501 19:55:53.262411 17747 leader_election.cc:302] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a0381278c204add884696237c5baf99; no voters:
I20240501 19:55:53.262691 17747 leader_election.cc:288] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:53.262775 17752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:53.262985 17752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [term 1 LEADER]: Becoming Leader. State: Replica: 0a0381278c204add884696237c5baf99, State: Running, Role: LEADER
I20240501 19:55:53.263314 17752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0381278c204add884696237c5baf99" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 46087 } }
I20240501 19:55:53.263759 17747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:53.265026 17754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0381278c204add884696237c5baf99. Latest consensus state: current_term: 1 leader_uuid: "0a0381278c204add884696237c5baf99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0381278c204add884696237c5baf99" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 46087 } } }
I20240501 19:55:53.265271 17754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:53.265720 17763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:53.266479 17753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a0381278c204add884696237c5baf99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0381278c204add884696237c5baf99" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 46087 } } }
I20240501 19:55:53.266645 17753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:53.269217 17763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:53.274312 17763 catalog_manager.cc:1347] Generated new cluster ID: aeccfe0d5520414bbbba52bf779dc8ce
I20240501 19:55:53.274380 17763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:53.286213 17763 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:53.287411 17763 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:53.299471 17763 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0a0381278c204add884696237c5baf99: Generated new TSK 0
I20240501 19:55:53.300305 17763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:53.308569 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.1:0
--local_ip_for_outbound_sockets=127.17.65.1
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--builtin_ntp_servers=127.17.65.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 19:55:53.413548 17771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:53.413738 17771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:53.413805 17771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:53.416743 17771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:53.416859 17771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.1
I20240501 19:55:53.420135 17771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--never_fsync=true
--heap_profile_path=/tmp/kudu.17771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.1
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:53.420912 17771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:53.421931 17771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:53.429947 17777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.430488 17779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.431412 17776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:53.432212 17771 server_base.cc:1034] running on GCE node
I20240501 19:55:53.432530 17771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:53.433012 17771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:53.434147 17771 hybrid_clock.cc:648] HybridClock initialized: now 1714593353434131 us; error 29 us; skew 500 ppm
I20240501 19:55:53.435861 17771 webserver.cc:460] Webserver started at http://127.17.65.1:35541/ using document root <none> and password file <none>
I20240501 19:55:53.436390 17771 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:53.436443 17771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:53.436600 17771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:53.438079 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e35921814be04c10827ff66f38b6be49"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.438510 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e35921814be04c10827ff66f38b6be49"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.441681 17771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:53.443603 17785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.444473 17771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:53.444571 17771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e35921814be04c10827ff66f38b6be49"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.444665 17771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:53.471127 17771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:53.471798 17771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:53.471971 17771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:53.472560 17771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:53.473467 17771 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:53.473537 17771 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.473605 17771 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:53.473647 17771 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.482255 17771 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.1:35339
I20240501 19:55:53.482282 17898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.1:35339 every 8 connection(s)
I20240501 19:55:53.483196 17771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:55:53.484314 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 17771
I20240501 19:55:53.484395 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:55:53.486590 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.2:0
--local_ip_for_outbound_sockets=127.17.65.2
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--builtin_ntp_servers=127.17.65.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 19:55:53.492233 17899 heartbeater.cc:347] Connected to a master server at 127.17.65.62:46087
I20240501 19:55:53.492470 17899 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:53.493099 17899 heartbeater.cc:511] Master 127.17.65.62:46087 requested a full tablet report, sending...
I20240501 19:55:53.494616 17711 ts_manager.cc:194] Registered new tserver with Master: e35921814be04c10827ff66f38b6be49 (127.17.65.1:35339)
I20240501 19:55:53.495774 17711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.1:33727
W20240501 19:55:53.583981 17902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:53.584148 17902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:53.584183 17902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:53.587074 17902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:53.587177 17902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.2
I20240501 19:55:53.590273 17902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--never_fsync=true
--heap_profile_path=/tmp/kudu.17902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.2
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:53.591025 17902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:53.592022 17902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:53.599272 17907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.599977 17910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.600826 17908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:53.601639 17902 server_base.cc:1034] running on GCE node
I20240501 19:55:53.601965 17902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:53.602442 17902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:53.603577 17902 hybrid_clock.cc:648] HybridClock initialized: now 1714593353603565 us; error 26 us; skew 500 ppm
I20240501 19:55:53.605212 17902 webserver.cc:460] Webserver started at http://127.17.65.2:38351/ using document root <none> and password file <none>
I20240501 19:55:53.605701 17902 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:53.605759 17902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:53.605931 17902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:53.607486 17902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "42f832ec6e15437f98271663b46cc4fc"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.607909 17902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "42f832ec6e15437f98271663b46cc4fc"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.610877 17902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:53.612740 17916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.613587 17902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:53.613679 17902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "42f832ec6e15437f98271663b46cc4fc"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.613749 17902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:53.629006 17902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:53.629531 17902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:53.629657 17902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:53.630101 17902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:53.630827 17902 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:53.630869 17902 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.630918 17902 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:53.630940 17902 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.638106 17902 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.2:41515
I20240501 19:55:53.638130 18029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.2:41515 every 8 connection(s)
I20240501 19:55:53.638833 17902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:55:53.641656 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 17902
I20240501 19:55:53.641767 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:55:53.643720 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.3:0
--local_ip_for_outbound_sockets=127.17.65.3
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--builtin_ntp_servers=127.17.65.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 19:55:53.646736 18030 heartbeater.cc:347] Connected to a master server at 127.17.65.62:46087
I20240501 19:55:53.646906 18030 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:53.647378 18030 heartbeater.cc:511] Master 127.17.65.62:46087 requested a full tablet report, sending...
I20240501 19:55:53.648268 17711 ts_manager.cc:194] Registered new tserver with Master: 42f832ec6e15437f98271663b46cc4fc (127.17.65.2:41515)
I20240501 19:55:53.648793 17711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.2:47943
W20240501 19:55:53.739904 18033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:53.740068 18033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:53.740113 18033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:53.742980 18033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:53.743122 18033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.3
I20240501 19:55:53.746361 18033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--never_fsync=true
--heap_profile_path=/tmp/kudu.18033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.3
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:53.747099 18033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:53.748095 18033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:53.755594 18039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.756513 18041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.757194 18038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:53.757975 18033 server_base.cc:1034] running on GCE node
I20240501 19:55:53.758361 18033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:53.758879 18033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:53.760012 18033 hybrid_clock.cc:648] HybridClock initialized: now 1714593353760002 us; error 25 us; skew 500 ppm
I20240501 19:55:53.761499 18033 webserver.cc:460] Webserver started at http://127.17.65.3:37365/ using document root <none> and password file <none>
I20240501 19:55:53.761955 18033 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:53.762008 18033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:53.762172 18033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:53.763643 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "28715eb1bc09440b844d07fb986c087b"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.764075 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "28715eb1bc09440b844d07fb986c087b"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.767213 18033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:53.769033 18047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.769815 18033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:53.769902 18033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "28715eb1bc09440b844d07fb986c087b"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.769968 18033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:53.791977 18033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:53.792482 18033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:53.792644 18033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:53.793061 18033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:53.793793 18033 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:53.793840 18033 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.793886 18033 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:53.793915 18033 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.800845 18033 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.3:36627
I20240501 19:55:53.800865 18160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.3:36627 every 8 connection(s)
I20240501 19:55:53.801612 18033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:55:53.808622 18161 heartbeater.cc:347] Connected to a master server at 127.17.65.62:46087
I20240501 19:55:53.808789 18161 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:53.809206 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 18033
I20240501 19:55:53.809273 18161 heartbeater.cc:511] Master 127.17.65.62:46087 requested a full tablet report, sending...
I20240501 19:55:53.809312 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:55:53.810168 17711 ts_manager.cc:194] Registered new tserver with Master: 28715eb1bc09440b844d07fb986c087b (127.17.65.3:36627)
I20240501 19:55:53.810643 17711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.3:42983
I20240501 19:55:53.811298 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.4:0
--local_ip_for_outbound_sockets=127.17.65.4
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--builtin_ntp_servers=127.17.65.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 19:55:53.909597 18164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:53.909771 18164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:53.909834 18164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:53.912660 18164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:53.912786 18164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.4
I20240501 19:55:53.916432 18164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--never_fsync=true
--heap_profile_path=/tmp/kudu.18164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.4
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:53.917222 18164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:53.918294 18164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:53.926399 18169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.926950 18170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.927899 18172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:53.928694 18164 server_base.cc:1034] running on GCE node
I20240501 19:55:53.929018 18164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:53.929489 18164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:53.930640 18164 hybrid_clock.cc:648] HybridClock initialized: now 1714593353930625 us; error 31 us; skew 500 ppm
I20240501 19:55:53.932312 18164 webserver.cc:460] Webserver started at http://127.17.65.4:43779/ using document root <none> and password file <none>
I20240501 19:55:53.932809 18164 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:53.932865 18164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:53.933015 18164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:53.934463 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "87b3d5224a404e28a6d6d265f251dadf"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.934880 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "87b3d5224a404e28a6d6d265f251dadf"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.937965 18164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 19:55:53.939854 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.940702 18164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:53.940793 18164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "87b3d5224a404e28a6d6d265f251dadf"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-703j"
I20240501 19:55:53.940866 18164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:53.958350 18164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:53.958911 18164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:53.959089 18164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:53.959547 18164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:53.960307 18164 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:53.960350 18164 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.960403 18164 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:53.960427 18164 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.968243 18164 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.4:33019
I20240501 19:55:53.968267 18291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.4:33019 every 8 connection(s)
I20240501 19:55:53.969018 18164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:55:53.976409 18292 heartbeater.cc:347] Connected to a master server at 127.17.65.62:46087
I20240501 19:55:53.976589 18292 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:53.977012 18292 heartbeater.cc:511] Master 127.17.65.62:46087 requested a full tablet report, sending...
I20240501 19:55:53.977824 17711 ts_manager.cc:194] Registered new tserver with Master: 87b3d5224a404e28a6d6d265f251dadf (127.17.65.4:33019)
I20240501 19:55:53.978332 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 18164
I20240501 19:55:53.978425 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:55:53.978439 17711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.4:51145
I20240501 19:55:53.980170 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.5:0
--local_ip_for_outbound_sockets=127.17.65.5
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--builtin_ntp_servers=127.17.65.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 19:55:54.079742 18295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:54.079908 18295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:54.079943 18295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:54.082849 18295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:54.082942 18295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.5
I20240501 19:55:54.086302 18295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:46087
--never_fsync=true
--heap_profile_path=/tmp/kudu.18295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.5
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:54.087070 18295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:54.088042 18295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:54.095736 18303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.096349 18301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.097247 18300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:54.098031 18295 server_base.cc:1034] running on GCE node
I20240501 19:55:54.098325 18295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:54.098774 18295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:54.099929 18295 hybrid_clock.cc:648] HybridClock initialized: now 1714593354099921 us; error 23 us; skew 500 ppm
I20240501 19:55:54.101409 18295 webserver.cc:460] Webserver started at http://127.17.65.5:46129/ using document root <none> and password file <none>
I20240501 19:55:54.101871 18295 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:54.101924 18295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:54.102068 18295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:54.103505 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "33925577ff434e43bcafd5a8249248b9"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.103910 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "33925577ff434e43bcafd5a8249248b9"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.106889 18295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:54.109079 18309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.109920 18295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:54.110014 18295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "33925577ff434e43bcafd5a8249248b9"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.110080 18295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:54.126538 18295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:54.127065 18295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:54.127199 18295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:54.127657 18295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:54.128386 18295 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:54.128428 18295 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.128477 18295 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:54.128500 18295 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.135780 18295 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.5:38993
I20240501 19:55:54.135821 18422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.5:38993 every 8 connection(s)
I20240501 19:55:54.136600 18295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:55:54.144057 18423 heartbeater.cc:347] Connected to a master server at 127.17.65.62:46087
I20240501 19:55:54.144245 18423 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:54.144676 18423 heartbeater.cc:511] Master 127.17.65.62:46087 requested a full tablet report, sending...
I20240501 19:55:54.144871 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 18295
I20240501 19:55:54.144953 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:55:54.145561 17711 ts_manager.cc:194] Registered new tserver with Master: 33925577ff434e43bcafd5a8249248b9 (127.17.65.5:38993)
I20240501 19:55:54.146121 17711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.5:50993
I20240501 19:55:54.147305 17668 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:55:54.157493 17711 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46766:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:55:54.173483 18095 tablet_service.cc:1467] Processing CreateTablet for tablet 26f9e96b6381488ab8de899cc7b8af22 (DEFAULT_TABLE table=TestTable [id=3af5274d1e554fe8bfea91ad0e84f3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:54.173491 17964 tablet_service.cc:1467] Processing CreateTablet for tablet 26f9e96b6381488ab8de899cc7b8af22 (DEFAULT_TABLE table=TestTable [id=3af5274d1e554fe8bfea91ad0e84f3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:54.174299 18095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f9e96b6381488ab8de899cc7b8af22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:54.174300 17964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f9e96b6381488ab8de899cc7b8af22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:54.177044 18357 tablet_service.cc:1467] Processing CreateTablet for tablet 26f9e96b6381488ab8de899cc7b8af22 (DEFAULT_TABLE table=TestTable [id=3af5274d1e554fe8bfea91ad0e84f3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:54.177805 18357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f9e96b6381488ab8de899cc7b8af22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:54.178781 18440 tablet_bootstrap.cc:492] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b: Bootstrap starting.
I20240501 19:55:54.179915 18441 tablet_bootstrap.cc:492] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc: Bootstrap starting.
I20240501 19:55:54.180495 18440 tablet_bootstrap.cc:654] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:54.181047 18440 log.cc:824] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:54.181691 18441 tablet_bootstrap.cc:654] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:54.182385 18441 log.cc:824] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:54.182526 18440 tablet_bootstrap.cc:492] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b: No bootstrap required, opened a new log
I20240501 19:55:54.182687 18440 ts_tablet_manager.cc:1367] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:55:54.183200 18444 tablet_bootstrap.cc:492] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9: Bootstrap starting.
I20240501 19:55:54.183938 18441 tablet_bootstrap.cc:492] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc: No bootstrap required, opened a new log
I20240501 19:55:54.184121 18441 ts_tablet_manager.cc:1367] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:55:54.184710 18444 tablet_bootstrap.cc:654] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:54.185236 18444 log.cc:824] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:54.186414 18444 tablet_bootstrap.cc:492] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9: No bootstrap required, opened a new log
I20240501 19:55:54.186559 18444 ts_tablet_manager.cc:1367] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:55:54.186506 18440 raft_consensus.cc:348] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.186707 18440 raft_consensus.cc:374] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:54.186785 18440 raft_consensus.cc:729] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28715eb1bc09440b844d07fb986c087b, State: Initialized, Role: FOLLOWER
I20240501 19:55:54.187211 18440 consensus_queue.cc:260] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.187230 18441 raft_consensus.cc:348] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.187391 18441 raft_consensus.cc:374] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:54.187480 18441 raft_consensus.cc:729] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f832ec6e15437f98271663b46cc4fc, State: Initialized, Role: FOLLOWER
I20240501 19:55:54.187889 18161 heartbeater.cc:503] Master 127.17.65.62:46087 was elected leader, sending a full tablet report...
I20240501 19:55:54.187871 18441 consensus_queue.cc:260] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.187999 18440 ts_tablet_manager.cc:1398] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:55:54.188445 18441 ts_tablet_manager.cc:1398] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:55:54.188700 18030 heartbeater.cc:503] Master 127.17.65.62:46087 was elected leader, sending a full tablet report...
I20240501 19:55:54.189682 18444 raft_consensus.cc:348] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.189844 18444 raft_consensus.cc:374] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:54.189908 18444 raft_consensus.cc:729] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33925577ff434e43bcafd5a8249248b9, State: Initialized, Role: FOLLOWER
I20240501 19:55:54.190253 18444 consensus_queue.cc:260] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.190939 18423 heartbeater.cc:503] Master 127.17.65.62:46087 was elected leader, sending a full tablet report...
I20240501 19:55:54.190986 18444 ts_tablet_manager.cc:1398] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:55:54.192428 18448 raft_consensus.cc:482] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:55:54.192556 18448 raft_consensus.cc:504] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.193622 18448 leader_election.cc:288] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42f832ec6e15437f98271663b46cc4fc (127.17.65.2:41515), 28715eb1bc09440b844d07fb986c087b (127.17.65.3:36627)
I20240501 19:55:54.197623 17984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f9e96b6381488ab8de899cc7b8af22" candidate_uuid: "33925577ff434e43bcafd5a8249248b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f832ec6e15437f98271663b46cc4fc" is_pre_election: true
I20240501 19:55:54.197649 18115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f9e96b6381488ab8de899cc7b8af22" candidate_uuid: "33925577ff434e43bcafd5a8249248b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28715eb1bc09440b844d07fb986c087b" is_pre_election: true
I20240501 19:55:54.197835 18115 raft_consensus.cc:2454] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33925577ff434e43bcafd5a8249248b9 in term 0.
I20240501 19:55:54.197908 17984 raft_consensus.cc:2454] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33925577ff434e43bcafd5a8249248b9 in term 0.
I20240501 19:55:54.198264 18313 leader_election.cc:302] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28715eb1bc09440b844d07fb986c087b, 33925577ff434e43bcafd5a8249248b9; no voters:
I20240501 19:55:54.198524 18448 raft_consensus.cc:2781] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:55:54.198601 18448 raft_consensus.cc:482] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:55:54.198630 18448 raft_consensus.cc:3037] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:54.199384 18448 raft_consensus.cc:504] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.199694 18448 leader_election.cc:288] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [CANDIDATE]: Term 1 election: Requested vote from peers 42f832ec6e15437f98271663b46cc4fc (127.17.65.2:41515), 28715eb1bc09440b844d07fb986c087b (127.17.65.3:36627)
I20240501 19:55:54.199947 17984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f9e96b6381488ab8de899cc7b8af22" candidate_uuid: "33925577ff434e43bcafd5a8249248b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f832ec6e15437f98271663b46cc4fc"
I20240501 19:55:54.200009 18115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f9e96b6381488ab8de899cc7b8af22" candidate_uuid: "33925577ff434e43bcafd5a8249248b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28715eb1bc09440b844d07fb986c087b"
I20240501 19:55:54.200047 17984 raft_consensus.cc:3037] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:54.200100 18115 raft_consensus.cc:3037] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:54.201028 17984 raft_consensus.cc:2454] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33925577ff434e43bcafd5a8249248b9 in term 1.
I20240501 19:55:54.201028 18115 raft_consensus.cc:2454] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33925577ff434e43bcafd5a8249248b9 in term 1.
I20240501 19:55:54.201330 18310 leader_election.cc:302] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33925577ff434e43bcafd5a8249248b9, 42f832ec6e15437f98271663b46cc4fc; no voters:
I20240501 19:55:54.201488 18448 raft_consensus.cc:2781] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:54.201653 18448 raft_consensus.cc:686] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [term 1 LEADER]: Becoming Leader. State: Replica: 33925577ff434e43bcafd5a8249248b9, State: Running, Role: LEADER
I20240501 19:55:54.201906 18448 consensus_queue.cc:237] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } }
I20240501 19:55:54.204227 17709 catalog_manager.cc:5484] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33925577ff434e43bcafd5a8249248b9 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "33925577ff434e43bcafd5a8249248b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f832ec6e15437f98271663b46cc4fc" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 41515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28715eb1bc09440b844d07fb986c087b" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33925577ff434e43bcafd5a8249248b9" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 38993 } health_report { overall_health: HEALTHY } } }
I20240501 19:55:54.218362 17668 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:55:54.219764 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e35921814be04c10827ff66f38b6be49 to finish bootstrapping
I20240501 19:55:54.225296 17668 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 42f832ec6e15437f98271663b46cc4fc to finish bootstrapping
I20240501 19:55:54.229455 17668 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 28715eb1bc09440b844d07fb986c087b to finish bootstrapping
I20240501 19:55:54.233301 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 87b3d5224a404e28a6d6d265f251dadf to finish bootstrapping
I20240501 19:55:54.238343 17668 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 33925577ff434e43bcafd5a8249248b9 to finish bootstrapping
W20240501 19:55:54.302518 18162 tablet.cc:2369] T 26f9e96b6381488ab8de899cc7b8af22 P 28715eb1bc09440b844d07fb986c087b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:55:54.387318 18424 tablet.cc:2369] T 26f9e96b6381488ab8de899cc7b8af22 P 33925577ff434e43bcafd5a8249248b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:55:54.389984 18031 tablet.cc:2369] T 26f9e96b6381488ab8de899cc7b8af22 P 42f832ec6e15437f98271663b46cc4fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:55:54.419478 18095 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.421927 18226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.422734 17833 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.422737 17964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.422816 18357 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:54.498154 17899 heartbeater.cc:503] Master 127.17.65.62:46087 was elected leader, sending a full tablet report...
I20240501 19:55:54.506831 18095 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.506902 17964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.509929 18226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.510429 17833 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.512280 18357 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:54.544926 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 17771
I20240501 19:55:54.553341 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 17902
I20240501 19:55:54.558828 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 18033
I20240501 19:55:54.564031 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 18164
I20240501 19:55:54.569083 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 18295
I20240501 19:55:54.574088 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 17680
2024-05-01T19:55:54Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (1530 ms)
[----------] 1 test from AdminCliTest (1530 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240501 19:55:54.596276 17668 test_util.cc:268] Using random seed: -539639887
I20240501 19:55:54.597002 17668 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:55:54.597038 17668 ts_itest-base.cc:113] --------------
I20240501 19:55:54.597057 17668 ts_itest-base.cc:114] 5 tablet servers
I20240501 19:55:54.597074 17668 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:55:54.597100 17668 ts_itest-base.cc:116] --------------
2024-05-01T19:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:54Z Disabled control of system clock
I20240501 19:55:54.604219 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:38715
--webserver_interface=127.17.65.62
--webserver_port=0
--builtin_ntp_servers=127.17.65.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.65.62:38715
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 19:55:54.705715 18528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:54.705948 18528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:54.705989 18528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:54.708950 18528 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:54.709017 18528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:54.709036 18528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:54.709060 18528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:54.709084 18528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:54.712651 18528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.17.65.62:38715
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.65.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.18528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:54.713518 18528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:54.714587 18528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:54.722416 18534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.723085 18536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.723945 18533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:54.724784 18528 server_base.cc:1034] running on GCE node
I20240501 19:55:54.725283 18528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:54.726192 18528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:54.727347 18528 hybrid_clock.cc:648] HybridClock initialized: now 1714593354727328 us; error 29 us; skew 500 ppm
I20240501 19:55:54.729492 18528 webserver.cc:460] Webserver started at http://127.17.65.62:34759/ using document root <none> and password file <none>
I20240501 19:55:54.730017 18528 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:54.730070 18528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:54.730245 18528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:54.731899 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "60367384e19d404f859066f824003e95"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.732378 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "60367384e19d404f859066f824003e95"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.735476 18528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:54.737471 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.738343 18528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:54.738447 18528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
uuid: "60367384e19d404f859066f824003e95"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.738535 18528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:54.765254 18528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:54.765933 18528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:54.766091 18528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:54.772043 18528 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.62:38715
I20240501 19:55:54.772059 18594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.62:38715 every 8 connection(s)
I20240501 19:55:54.772935 18528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:55:54.775322 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:54.779371 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 18528
I20240501 19:55:54.779471 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:55:54.780031 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95: Bootstrap starting.
I20240501 19:55:54.782223 18595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:54.782876 18595 log.cc:824] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:54.784363 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95: No bootstrap required, opened a new log
I20240501 19:55:54.787070 18595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60367384e19d404f859066f824003e95" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 38715 } }
I20240501 19:55:54.787271 18595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:54.787329 18595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60367384e19d404f859066f824003e95, State: Initialized, Role: FOLLOWER
I20240501 19:55:54.787704 18595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60367384e19d404f859066f824003e95" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 38715 } }
I20240501 19:55:54.787799 18595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:54.787833 18595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:54.787904 18595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:54.788702 18595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60367384e19d404f859066f824003e95" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 38715 } }
I20240501 19:55:54.788972 18595 leader_election.cc:302] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60367384e19d404f859066f824003e95; no voters:
I20240501 19:55:54.789178 18595 leader_election.cc:288] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:54.789266 18600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:54.789469 18600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [term 1 LEADER]: Becoming Leader. State: Replica: 60367384e19d404f859066f824003e95, State: Running, Role: LEADER
I20240501 19:55:54.789734 18600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60367384e19d404f859066f824003e95" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 38715 } }
I20240501 19:55:54.790138 18595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:54.791164 18601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60367384e19d404f859066f824003e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60367384e19d404f859066f824003e95" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 38715 } } }
I20240501 19:55:54.791303 18601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:54.791164 18602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60367384e19d404f859066f824003e95. Latest consensus state: current_term: 1 leader_uuid: "60367384e19d404f859066f824003e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60367384e19d404f859066f824003e95" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 38715 } } }
I20240501 19:55:54.791458 18602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:54.791708 18608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:54.794482 18608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:54.799346 18608 catalog_manager.cc:1347] Generated new cluster ID: 089886029e6c4e1a9795b4e40a713f13
I20240501 19:55:54.799410 18608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:54.806826 18608 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:54.807665 18608 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:54.812769 18608 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 60367384e19d404f859066f824003e95: Generated new TSK 0
I20240501 19:55:54.813470 18608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:54.824103 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.1:0
--local_ip_for_outbound_sockets=127.17.65.1
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--builtin_ntp_servers=127.17.65.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:55:54.931196 18619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:54.931402 18619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:54.931468 18619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:54.934334 18619 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:54.934399 18619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:54.934489 18619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.1
I20240501 19:55:54.937953 18619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--never_fsync=true
--heap_profile_path=/tmp/kudu.18619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.1
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:54.938773 18619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:54.939882 18619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:54.948051 18624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.948907 18627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.949751 18625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:54.950476 18619 server_base.cc:1034] running on GCE node
I20240501 19:55:54.950815 18619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:54.951350 18619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:54.952507 18619 hybrid_clock.cc:648] HybridClock initialized: now 1714593354952495 us; error 23 us; skew 500 ppm
I20240501 19:55:54.954293 18619 webserver.cc:460] Webserver started at http://127.17.65.1:34645/ using document root <none> and password file <none>
I20240501 19:55:54.954809 18619 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:54.954867 18619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:54.955019 18619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:54.956657 18619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fb51a53258604a12825a177596699dd0"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.957085 18619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fb51a53258604a12825a177596699dd0"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.960188 18619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 19:55:54.962236 18633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.963141 18619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:54.963227 18619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fb51a53258604a12825a177596699dd0"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-703j"
I20240501 19:55:54.963310 18619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:54.981433 18619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:54.981985 18619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:54.982126 18619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:54.982604 18619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:54.983467 18619 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:54.983511 18619 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.983562 18619 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:54.983592 18619 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.991429 18619 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.1:37027
I20240501 19:55:54.991452 18746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.1:37027 every 8 connection(s)
I20240501 19:55:54.992367 18619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:55:55.000149 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 18619
I20240501 19:55:55.000245 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:55:55.000459 18747 heartbeater.cc:347] Connected to a master server at 127.17.65.62:38715
I20240501 19:55:55.000664 18747 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:55.001209 18747 heartbeater.cc:511] Master 127.17.65.62:38715 requested a full tablet report, sending...
I20240501 19:55:55.002214 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.2:0
--local_ip_for_outbound_sockets=127.17.65.2
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--builtin_ntp_servers=127.17.65.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:55.002647 18559 ts_manager.cc:194] Registered new tserver with Master: fb51a53258604a12825a177596699dd0 (127.17.65.1:37027)
I20240501 19:55:55.004177 18559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.1:41153
W20240501 19:55:55.102471 18750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:55.102653 18750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:55.102695 18750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:55.105646 18750 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:55.105701 18750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:55.105803 18750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.2
I20240501 19:55:55.109308 18750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--never_fsync=true
--heap_profile_path=/tmp/kudu.18750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.2
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:55.110081 18750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:55.111131 18750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:55.119314 18758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:55.120245 18755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:55.120908 18756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:55.121610 18750 server_base.cc:1034] running on GCE node
I20240501 19:55:55.121968 18750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:55.122473 18750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:55.123620 18750 hybrid_clock.cc:648] HybridClock initialized: now 1714593355123611 us; error 28 us; skew 500 ppm
I20240501 19:55:55.125362 18750 webserver.cc:460] Webserver started at http://127.17.65.2:45473/ using document root <none> and password file <none>
I20240501 19:55:55.125887 18750 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:55.125945 18750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:55.126106 18750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:55.127784 18750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "665beaf1fb6a455ea491091ea5f62268"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.128211 18750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "665beaf1fb6a455ea491091ea5f62268"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.131335 18750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:55.133304 18764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.134195 18750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:55.134299 18750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "665beaf1fb6a455ea491091ea5f62268"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.134370 18750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:55.164068 18750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:55.164683 18750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:55.164837 18750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:55.165382 18750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:55.166282 18750 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:55.166328 18750 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.166396 18750 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:55.166424 18750 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.174672 18750 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.2:33449
I20240501 19:55:55.174706 18877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.2:33449 every 8 connection(s)
I20240501 19:55:55.175593 18750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:55:55.178296 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 18750
I20240501 19:55:55.178393 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:55:55.180189 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.3:0
--local_ip_for_outbound_sockets=127.17.65.3
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--builtin_ntp_servers=127.17.65.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:55.183702 18878 heartbeater.cc:347] Connected to a master server at 127.17.65.62:38715
I20240501 19:55:55.183874 18878 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:55.184304 18878 heartbeater.cc:511] Master 127.17.65.62:38715 requested a full tablet report, sending...
I20240501 19:55:55.185230 18559 ts_manager.cc:194] Registered new tserver with Master: 665beaf1fb6a455ea491091ea5f62268 (127.17.65.2:33449)
I20240501 19:55:55.186039 18559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.2:54871
W20240501 19:55:55.283653 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:55.283854 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:55.283900 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:55.286849 18881 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:55.286908 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:55.286976 18881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.3
I20240501 19:55:55.290241 18881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--never_fsync=true
--heap_profile_path=/tmp/kudu.18881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.3
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:55.290966 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:55.292013 18881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:55.300021 18886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:55.300603 18887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:55.301263 18889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:55.302232 18881 server_base.cc:1034] running on GCE node
I20240501 19:55:55.302584 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:55.303078 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:55.304234 18881 hybrid_clock.cc:648] HybridClock initialized: now 1714593355304224 us; error 24 us; skew 500 ppm
I20240501 19:55:55.305914 18881 webserver.cc:460] Webserver started at http://127.17.65.3:40247/ using document root <none> and password file <none>
I20240501 19:55:55.306445 18881 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:55.306504 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:55.306660 18881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:55.308199 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9f3b10f290414f52860036d866258ec1"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.308671 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9f3b10f290414f52860036d866258ec1"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.311700 18881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 19:55:55.313721 18895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.314661 18881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:55.314764 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9f3b10f290414f52860036d866258ec1"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.314841 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:55.347754 18881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:55.348320 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:55.348462 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:55.348979 18881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:55.349826 18881 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:55.349870 18881 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.349921 18881 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:55.349949 18881 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.357484 18881 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.3:38511
I20240501 19:55:55.357523 19008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.3:38511 every 8 connection(s)
I20240501 19:55:55.358332 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:55:55.365898 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 18881
I20240501 19:55:55.365988 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:55:55.366395 19009 heartbeater.cc:347] Connected to a master server at 127.17.65.62:38715
I20240501 19:55:55.366591 19009 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:55.367069 19009 heartbeater.cc:511] Master 127.17.65.62:38715 requested a full tablet report, sending...
I20240501 19:55:55.367867 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.4:0
--local_ip_for_outbound_sockets=127.17.65.4
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--builtin_ntp_servers=127.17.65.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:55.367918 18559 ts_manager.cc:194] Registered new tserver with Master: 9f3b10f290414f52860036d866258ec1 (127.17.65.3:38511)
I20240501 19:55:55.368443 18559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.3:41283
W20240501 19:55:55.468327 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:55.468520 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:55.468569 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:55.471450 19012 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:55.471504 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:55.471601 19012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.4
I20240501 19:55:55.474892 19012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--never_fsync=true
--heap_profile_path=/tmp/kudu.19012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.4
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:55.475679 19012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:55.476707 19012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:55.484934 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:55.485888 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:55.486697 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:55.487398 19012 server_base.cc:1034] running on GCE node
I20240501 19:55:55.487735 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:55.488200 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:55.489333 19012 hybrid_clock.cc:648] HybridClock initialized: now 1714593355489318 us; error 29 us; skew 500 ppm
I20240501 19:55:55.490994 19012 webserver.cc:460] Webserver started at http://127.17.65.4:43497/ using document root <none> and password file <none>
I20240501 19:55:55.491519 19012 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:55.491573 19012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:55.491752 19012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:55.493314 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.493784 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.496950 19012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:55:55.498840 19026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.499748 19012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:55.499835 19012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.499924 19012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:55.513913 19012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:55.514410 19012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:55.514550 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:55.515000 19012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:55.515834 19012 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:55.515880 19012 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.515936 19012 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:55.515961 19012 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.523861 19012 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.4:44125
I20240501 19:55:55.523885 19139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.4:44125 every 8 connection(s)
I20240501 19:55:55.524708 19012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:55:55.532697 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 19012
I20240501 19:55:55.532804 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:55:55.533883 19140 heartbeater.cc:347] Connected to a master server at 127.17.65.62:38715
I20240501 19:55:55.534121 19140 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:55.534644 19140 heartbeater.cc:511] Master 127.17.65.62:38715 requested a full tablet report, sending...
I20240501 19:55:55.536042 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.5:0
--local_ip_for_outbound_sockets=127.17.65.5
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--builtin_ntp_servers=127.17.65.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:55.536083 18559 ts_manager.cc:194] Registered new tserver with Master: 2a8de3ca506a4e3a8ef640c5fd234dfa (127.17.65.4:44125)
I20240501 19:55:55.536705 18559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.4:57551
W20240501 19:55:55.637423 19143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:55.637629 19143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:55.637672 19143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:55.640600 19143 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:55.640661 19143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:55.640758 19143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.5
I20240501 19:55:55.644306 19143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:38715
--never_fsync=true
--heap_profile_path=/tmp/kudu.19143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.5
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:55.645118 19143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:55.646176 19143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:55.654646 19151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:55.655529 19149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:55.656215 19148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:55.656946 19143 server_base.cc:1034] running on GCE node
I20240501 19:55:55.657305 19143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:55.657822 19143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:55.658959 19143 hybrid_clock.cc:648] HybridClock initialized: now 1714593355658944 us; error 28 us; skew 500 ppm
I20240501 19:55:55.660766 19143 webserver.cc:460] Webserver started at http://127.17.65.5:38485/ using document root <none> and password file <none>
I20240501 19:55:55.661298 19143 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:55.661366 19143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:55.661528 19143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:55.663123 19143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "53440b63d998489eabd815489c8c3567"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.663558 19143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "53440b63d998489eabd815489c8c3567"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.666718 19143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:55.668754 19157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.669637 19143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 19:55:55.669732 19143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "53440b63d998489eabd815489c8c3567"
format_stamp: "Formatted at 2024-05-01 19:55:55 on dist-test-slave-703j"
I20240501 19:55:55.669806 19143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:55.714395 19143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:55.714973 19143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:55.715170 19143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:55.715708 19143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:55.716569 19143 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:55.716625 19143 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.716678 19143 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:55.716703 19143 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:55.724623 19143 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.5:44577
I20240501 19:55:55.724670 19270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.5:44577 every 8 connection(s)
I20240501 19:55:55.725502 19143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:55:55.732767 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 19143
I20240501 19:55:55.732875 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:55:55.733127 19271 heartbeater.cc:347] Connected to a master server at 127.17.65.62:38715
I20240501 19:55:55.733335 19271 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:55.733803 19271 heartbeater.cc:511] Master 127.17.65.62:38715 requested a full tablet report, sending...
I20240501 19:55:55.734660 18558 ts_manager.cc:194] Registered new tserver with Master: 53440b63d998489eabd815489c8c3567 (127.17.65.5:44577)
I20240501 19:55:55.735190 18558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.5:60679
I20240501 19:55:55.746332 17668 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:55:55.754244 18558 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42020:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:55:55.770494 19205 tablet_service.cc:1467] Processing CreateTablet for tablet 65f76759b3f64fd896c35f2c3388fed3 (DEFAULT_TABLE table=TestTable [id=895b98ceccde43a8b3bf425c3506e574]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:55.770573 19074 tablet_service.cc:1467] Processing CreateTablet for tablet 65f76759b3f64fd896c35f2c3388fed3 (DEFAULT_TABLE table=TestTable [id=895b98ceccde43a8b3bf425c3506e574]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:55.770574 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 65f76759b3f64fd896c35f2c3388fed3 (DEFAULT_TABLE table=TestTable [id=895b98ceccde43a8b3bf425c3506e574]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:55.771456 19205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f76759b3f64fd896c35f2c3388fed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:55.771456 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f76759b3f64fd896c35f2c3388fed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:55.771456 19074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f76759b3f64fd896c35f2c3388fed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:55.777565 19289 tablet_bootstrap.cc:492] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa: Bootstrap starting.
I20240501 19:55:55.777565 19290 tablet_bootstrap.cc:492] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567: Bootstrap starting.
I20240501 19:55:55.779621 19290 tablet_bootstrap.cc:654] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:55.779621 19289 tablet_bootstrap.cc:654] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:55.780360 19289 log.cc:824] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:55.780360 19290 log.cc:824] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:55.780928 19288 tablet_bootstrap.cc:492] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268: Bootstrap starting.
I20240501 19:55:55.782308 19289 tablet_bootstrap.cc:492] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa: No bootstrap required, opened a new log
I20240501 19:55:55.782476 19289 ts_tablet_manager.cc:1367] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:55:55.782883 19288 tablet_bootstrap.cc:654] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:55.783636 19288 log.cc:824] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:55.784857 19290 tablet_bootstrap.cc:492] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567: No bootstrap required, opened a new log
I20240501 19:55:55.785018 19290 ts_tablet_manager.cc:1367] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 19:55:55.785295 19288 tablet_bootstrap.cc:492] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268: No bootstrap required, opened a new log
I20240501 19:55:55.785461 19288 ts_tablet_manager.cc:1367] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 19:55:55.786188 19289 raft_consensus.cc:348] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:55.786425 19289 raft_consensus.cc:374] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:55.786479 19289 raft_consensus.cc:729] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8de3ca506a4e3a8ef640c5fd234dfa, State: Initialized, Role: FOLLOWER
I20240501 19:55:55.786942 19289 consensus_queue.cc:260] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:55.787751 19140 heartbeater.cc:503] Master 127.17.65.62:38715 was elected leader, sending a full tablet report...
I20240501 19:55:55.788378 19288 raft_consensus.cc:348] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:55.788548 19288 raft_consensus.cc:374] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:55.788599 19288 raft_consensus.cc:729] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665beaf1fb6a455ea491091ea5f62268, State: Initialized, Role: FOLLOWER
I20240501 19:55:55.788638 19290 raft_consensus.cc:348] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:55.788817 19290 raft_consensus.cc:374] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:55.788868 19290 raft_consensus.cc:729] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53440b63d998489eabd815489c8c3567, State: Initialized, Role: FOLLOWER
I20240501 19:55:55.789078 19289 ts_tablet_manager.cc:1398] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 19:55:55.789216 19290 consensus_queue.cc:260] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:55.789748 19271 heartbeater.cc:503] Master 127.17.65.62:38715 was elected leader, sending a full tablet report...
I20240501 19:55:55.789911 19290 ts_tablet_manager.cc:1398] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:55:55.790493 19288 consensus_queue.cc:260] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:55.791152 18878 heartbeater.cc:503] Master 127.17.65.62:38715 was elected leader, sending a full tablet report...
I20240501 19:55:55.791379 19288 ts_tablet_manager.cc:1398] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20240501 19:55:55.926802 18879 tablet.cc:2369] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:55:55.976341 19272 tablet.cc:2369] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:55:55.995841 19296 raft_consensus.cc:482] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:55:55.996026 19296 raft_consensus.cc:504] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:55.997130 19296 leader_election.cc:288] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a8de3ca506a4e3a8ef640c5fd234dfa (127.17.65.4:44125), 53440b63d998489eabd815489c8c3567 (127.17.65.5:44577)
I20240501 19:55:56.000726 19094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f76759b3f64fd896c35f2c3388fed3" candidate_uuid: "665beaf1fb6a455ea491091ea5f62268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" is_pre_election: true
I20240501 19:55:56.000800 19225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f76759b3f64fd896c35f2c3388fed3" candidate_uuid: "665beaf1fb6a455ea491091ea5f62268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53440b63d998489eabd815489c8c3567" is_pre_election: true
I20240501 19:55:56.000982 19094 raft_consensus.cc:2454] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665beaf1fb6a455ea491091ea5f62268 in term 0.
I20240501 19:55:56.001049 19225 raft_consensus.cc:2454] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665beaf1fb6a455ea491091ea5f62268 in term 0.
I20240501 19:55:56.001374 18765 leader_election.cc:302] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8de3ca506a4e3a8ef640c5fd234dfa, 665beaf1fb6a455ea491091ea5f62268; no voters:
I20240501 19:55:56.001608 19296 raft_consensus.cc:2781] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:55:56.001673 19296 raft_consensus.cc:482] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:55:56.001701 19296 raft_consensus.cc:3037] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:56.002606 19296 raft_consensus.cc:504] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:56.002920 19296 leader_election.cc:288] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [CANDIDATE]: Term 1 election: Requested vote from peers 2a8de3ca506a4e3a8ef640c5fd234dfa (127.17.65.4:44125), 53440b63d998489eabd815489c8c3567 (127.17.65.5:44577)
I20240501 19:55:56.003248 19225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f76759b3f64fd896c35f2c3388fed3" candidate_uuid: "665beaf1fb6a455ea491091ea5f62268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53440b63d998489eabd815489c8c3567"
I20240501 19:55:56.003365 19225 raft_consensus.cc:3037] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:56.004190 19225 raft_consensus.cc:2454] T 65f76759b3f64fd896c35f2c3388fed3 P 53440b63d998489eabd815489c8c3567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665beaf1fb6a455ea491091ea5f62268 in term 1.
I20240501 19:55:56.004403 19094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f76759b3f64fd896c35f2c3388fed3" candidate_uuid: "665beaf1fb6a455ea491091ea5f62268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa"
I20240501 19:55:56.004472 18768 leader_election.cc:302] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53440b63d998489eabd815489c8c3567, 665beaf1fb6a455ea491091ea5f62268; no voters:
I20240501 19:55:56.004499 19094 raft_consensus.cc:3037] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:56.004622 19296 raft_consensus.cc:2781] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:56.004792 19296 raft_consensus.cc:686] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [term 1 LEADER]: Becoming Leader. State: Replica: 665beaf1fb6a455ea491091ea5f62268, State: Running, Role: LEADER
I20240501 19:55:56.005103 19296 consensus_queue.cc:237] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } }
I20240501 19:55:56.005395 19094 raft_consensus.cc:2454] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665beaf1fb6a455ea491091ea5f62268 in term 1.
I20240501 19:55:56.006680 18747 heartbeater.cc:503] Master 127.17.65.62:38715 was elected leader, sending a full tablet report...
I20240501 19:55:56.007624 18559 catalog_manager.cc:5484] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 reported cstate change: term changed from 0 to 1, leader changed from <none> to 665beaf1fb6a455ea491091ea5f62268 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "665beaf1fb6a455ea491091ea5f62268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 } } peers { permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 } } peers { permanent_uuid: "665beaf1fb6a455ea491091ea5f62268" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 33449 } } }
W20240501 19:55:56.025636 19141 tablet.cc:2369] T 65f76759b3f64fd896c35f2c3388fed3 P 2a8de3ca506a4e3a8ef640c5fd234dfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:55:56.057941 17668 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:55:56.059116 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fb51a53258604a12825a177596699dd0 to finish bootstrapping
I20240501 19:55:56.070117 17668 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 665beaf1fb6a455ea491091ea5f62268 to finish bootstrapping
I20240501 19:55:56.074334 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9f3b10f290414f52860036d866258ec1 to finish bootstrapping
I20240501 19:55:56.079193 17668 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2a8de3ca506a4e3a8ef640c5fd234dfa to finish bootstrapping
I20240501 19:55:56.083333 17668 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 53440b63d998489eabd815489c8c3567 to finish bootstrapping
I20240501 19:55:56.087103 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 18619
I20240501 19:55:56.275988 19074 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.276162 18812 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.276538 19205 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.277638 18943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.347107 18943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.347585 19074 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.350307 19205 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.350560 18812 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.370122 19009 heartbeater.cc:503] Master 127.17.65.62:38715 was elected leader, sending a full tablet report...
I20240501 19:55:56.384390 19296 consensus_queue.cc:1035] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53440b63d998489eabd815489c8c3567" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 44577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:55:56.390079 19299 consensus_queue.cc:1035] T 65f76759b3f64fd896c35f2c3388fed3 P 665beaf1fb6a455ea491091ea5f62268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8de3ca506a4e3a8ef640c5fd234dfa" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 44125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:55:56.450798 18943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.451627 19074 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.451753 19205 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.452137 18812 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.656297 19074 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.657891 18943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.658022 18812 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.658236 19205 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.723153 19074 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.725101 19205 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.725678 18812 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.726994 18943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.821950 19074 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.822715 19205 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.823311 18943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.824131 18812 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.853516 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 18750
I20240501 19:55:56.859606 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 18881
I20240501 19:55:56.864938 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 19012
I20240501 19:55:56.869954 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 19143
I20240501 19:55:56.874998 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 18528
2024-05-01T19:55:56Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (2300 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2300 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240501 19:55:56.896790 17668 test_util.cc:268] Using random seed: -537339374
I20240501 19:55:56.897511 17668 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:55:56.897545 17668 ts_itest-base.cc:113] --------------
I20240501 19:55:56.897562 17668 ts_itest-base.cc:114] 7 tablet servers
I20240501 19:55:56.897578 17668 ts_itest-base.cc:115] 1 replicas per TS
I20240501 19:55:56.897603 17668 ts_itest-base.cc:116] --------------
2024-05-01T19:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:56Z Disabled control of system clock
I20240501 19:55:56.904830 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:37761
--webserver_interface=127.17.65.62
--webserver_port=0
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.65.62:37761
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 19:55:57.003480 19493 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 19:55:57.003703 19493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.003752 19493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.003816 19493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.006686 19493 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:57.006757 19493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:57.006800 19493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.006832 19493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:57.006865 19493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:57.010246 19493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.17.65.62:37761
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.65.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.19493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.011049 19493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.012027 19493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.019083 19498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.019671 19499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.020232 19501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.021210 19493 server_base.cc:1034] running on GCE node
I20240501 19:55:57.021595 19493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.022349 19493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.023478 19493 hybrid_clock.cc:648] HybridClock initialized: now 1714593357023461 us; error 32 us; skew 500 ppm
I20240501 19:55:57.025101 19493 webserver.cc:460] Webserver started at http://127.17.65.62:38077/ using document root <none> and password file <none>
I20240501 19:55:57.025625 19493 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.025678 19493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.025847 19493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.027431 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "def032d7b005429b87e9fe6941d0e6e6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.027870 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "def032d7b005429b87e9fe6941d0e6e6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.030841 19493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:57.032701 19507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.033572 19493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:57.033675 19493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
uuid: "def032d7b005429b87e9fe6941d0e6e6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.033746 19493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.049661 19493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.050262 19493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.050444 19493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.056705 19493 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.62:37761
I20240501 19:55:57.056718 19559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.62:37761 every 8 connection(s)
I20240501 19:55:57.057660 19493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:55:57.059486 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 19493
I20240501 19:55:57.059592 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:55:57.060094 19560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:57.065207 19560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6: Bootstrap starting.
I20240501 19:55:57.067497 19560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:57.068212 19560 log.cc:824] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:57.069578 19560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6: No bootstrap required, opened a new log
I20240501 19:55:57.071772 19560 raft_consensus.cc:348] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def032d7b005429b87e9fe6941d0e6e6" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 37761 } }
I20240501 19:55:57.071923 19560 raft_consensus.cc:374] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:57.071980 19560 raft_consensus.cc:729] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def032d7b005429b87e9fe6941d0e6e6, State: Initialized, Role: FOLLOWER
I20240501 19:55:57.072340 19560 consensus_queue.cc:260] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def032d7b005429b87e9fe6941d0e6e6" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 37761 } }
I20240501 19:55:57.072468 19560 raft_consensus.cc:388] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:57.072525 19560 raft_consensus.cc:482] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:57.072640 19560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:57.073367 19560 raft_consensus.cc:504] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def032d7b005429b87e9fe6941d0e6e6" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 37761 } }
I20240501 19:55:57.073681 19560 leader_election.cc:302] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: def032d7b005429b87e9fe6941d0e6e6; no voters:
I20240501 19:55:57.073931 19560 leader_election.cc:288] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:57.074267 19565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:57.074442 19565 raft_consensus.cc:686] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [term 1 LEADER]: Becoming Leader. State: Replica: def032d7b005429b87e9fe6941d0e6e6, State: Running, Role: LEADER
I20240501 19:55:57.074683 19565 consensus_queue.cc:237] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def032d7b005429b87e9fe6941d0e6e6" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 37761 } }
I20240501 19:55:57.074851 19560 sys_catalog.cc:564] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:57.076007 19566 sys_catalog.cc:455] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "def032d7b005429b87e9fe6941d0e6e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def032d7b005429b87e9fe6941d0e6e6" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 37761 } } }
I20240501 19:55:57.076115 19566 sys_catalog.cc:458] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:57.076467 19574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:57.076426 19567 sys_catalog.cc:455] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader def032d7b005429b87e9fe6941d0e6e6. Latest consensus state: current_term: 1 leader_uuid: "def032d7b005429b87e9fe6941d0e6e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def032d7b005429b87e9fe6941d0e6e6" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 37761 } } }
I20240501 19:55:57.077031 19567 sys_catalog.cc:458] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:57.078877 19574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:57.082981 19574 catalog_manager.cc:1347] Generated new cluster ID: e913a82ac01841c3bc2363154a141b0e
I20240501 19:55:57.083045 19574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:57.097628 19574 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:57.098549 19574 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:57.109555 19574 catalog_manager.cc:5857] T 00000000000000000000000000000000 P def032d7b005429b87e9fe6941d0e6e6: Generated new TSK 0
I20240501 19:55:57.110172 19574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:57.115248 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.1:0
--local_ip_for_outbound_sockets=127.17.65.1
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:55:57.213644 19584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.213850 19584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.213927 19584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.216953 19584 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:57.217025 19584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.217113 19584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.1
I20240501 19:55:57.220568 19584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.19584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.1
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.221405 19584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.222432 19584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.231132 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.232041 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.232827 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.233434 19584 server_base.cc:1034] running on GCE node
I20240501 19:55:57.233752 19584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.234237 19584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.235394 19584 hybrid_clock.cc:648] HybridClock initialized: now 1714593357235382 us; error 28 us; skew 500 ppm
I20240501 19:55:57.237089 19584 webserver.cc:460] Webserver started at http://127.17.65.1:37859/ using document root <none> and password file <none>
I20240501 19:55:57.237591 19584 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.237736 19584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.238008 19584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.239763 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "002bd5b7fe4c40169448f30e32ec54c6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.240260 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "002bd5b7fe4c40169448f30e32ec54c6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.243432 19584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:57.245664 19598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.246618 19584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.246717 19584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "002bd5b7fe4c40169448f30e32ec54c6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.246793 19584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.259107 19584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.259630 19584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.259763 19584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.260196 19584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.260957 19584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.260999 19584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.261050 19584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.261072 19584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.268769 19584 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.1:36451
I20240501 19:55:57.268798 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.1:36451 every 8 connection(s)
I20240501 19:55:57.269616 19584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:55:57.270422 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 19584
I20240501 19:55:57.270505 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:55:57.272521 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.2:0
--local_ip_for_outbound_sockets=127.17.65.2
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:57.278594 19712 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:55:57.278792 19712 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.279235 19712 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:55:57.280543 19524 ts_manager.cc:194] Registered new tserver with Master: 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:55:57.281637 19524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.1:34909
W20240501 19:55:57.375416 19715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.375636 19715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.375705 19715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.378556 19715 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:57.378616 19715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.378695 19715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.2
I20240501 19:55:57.381996 19715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.19715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.2
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.382786 19715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.383813 19715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.391793 19720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.392378 19723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.393312 19721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.394049 19715 server_base.cc:1034] running on GCE node
I20240501 19:55:57.394351 19715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.394788 19715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.395920 19715 hybrid_clock.cc:648] HybridClock initialized: now 1714593357395909 us; error 24 us; skew 500 ppm
I20240501 19:55:57.397584 19715 webserver.cc:460] Webserver started at http://127.17.65.2:33545/ using document root <none> and password file <none>
I20240501 19:55:57.398061 19715 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.398113 19715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.398272 19715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.399806 19715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1af1fefd33104e7bad976e2bd786c0b6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.400226 19715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1af1fefd33104e7bad976e2bd786c0b6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.403086 19715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:57.404973 19729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.405839 19715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.405937 19715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1af1fefd33104e7bad976e2bd786c0b6"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.406013 19715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.424075 19715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.424685 19715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.424824 19715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.425307 19715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.426143 19715 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.426187 19715 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.426230 19715 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.426260 19715 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.434298 19715 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.2:42021
I20240501 19:55:57.434324 19842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.2:42021 every 8 connection(s)
I20240501 19:55:57.435163 19715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:55:57.437664 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 19715
I20240501 19:55:57.437769 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:55:57.439636 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.3:0
--local_ip_for_outbound_sockets=127.17.65.3
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:57.443986 19843 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:55:57.444224 19843 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.444761 19843 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:55:57.445693 19524 ts_manager.cc:194] Registered new tserver with Master: 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:55:57.446203 19524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.2:36291
W20240501 19:55:57.541182 19846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.541399 19846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.541443 19846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.544396 19846 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:57.544459 19846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.544555 19846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.3
I20240501 19:55:57.548066 19846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.19846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.3
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.548920 19846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.550067 19846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.558315 19851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.558902 19852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.559556 19854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.560586 19846 server_base.cc:1034] running on GCE node
I20240501 19:55:57.560952 19846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.561518 19846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.562656 19846 hybrid_clock.cc:648] HybridClock initialized: now 1714593357562633 us; error 40 us; skew 500 ppm
I20240501 19:55:57.564455 19846 webserver.cc:460] Webserver started at http://127.17.65.3:39217/ using document root <none> and password file <none>
I20240501 19:55:57.565016 19846 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.565076 19846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.565259 19846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.566884 19846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b01a4a9860ce4bff90c3dc414228d44e"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.567378 19846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b01a4a9860ce4bff90c3dc414228d44e"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.570437 19846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:57.572409 19860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.573346 19846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:57.573443 19846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b01a4a9860ce4bff90c3dc414228d44e"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.573522 19846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.587391 19846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.587983 19846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.588129 19846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.588661 19846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.589497 19846 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.589542 19846 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.589596 19846 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.589623 19846 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.597445 19846 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.3:44125
I20240501 19:55:57.597482 19973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.3:44125 every 8 connection(s)
I20240501 19:55:57.598361 19846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:55:57.605692 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 19846
I20240501 19:55:57.605789 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:55:57.606168 19974 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:55:57.606387 19974 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.606850 19974 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:55:57.607779 19524 ts_manager.cc:194] Registered new tserver with Master: b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3:44125)
I20240501 19:55:57.607828 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.4:0
--local_ip_for_outbound_sockets=127.17.65.4
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:57.608352 19524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.3:36517
W20240501 19:55:57.709606 19977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.709818 19977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.709862 19977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.712783 19977 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:57.712846 19977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.712908 19977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.4
I20240501 19:55:57.716308 19977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.19977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.4
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.717099 19977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.718106 19977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.726301 19982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.727181 19983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.727870 19985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.728626 19977 server_base.cc:1034] running on GCE node
I20240501 19:55:57.728965 19977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.729489 19977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.730630 19977 hybrid_clock.cc:648] HybridClock initialized: now 1714593357730616 us; error 27 us; skew 500 ppm
I20240501 19:55:57.732333 19977 webserver.cc:460] Webserver started at http://127.17.65.4:37191/ using document root <none> and password file <none>
I20240501 19:55:57.732847 19977 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.732900 19977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.733072 19977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.734668 19977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2d6860caa69249c1837378d6f5247099"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.735144 19977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2d6860caa69249c1837378d6f5247099"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.738237 19977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:55:57.740200 19991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.741111 19977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 19:55:57.741206 19977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2d6860caa69249c1837378d6f5247099"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.741276 19977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.752468 19977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.753024 19977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.753166 19977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.753641 19977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.754467 19977 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.754511 19977 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.754573 19977 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.754598 19977 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.762094 19977 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.4:42703
I20240501 19:55:57.762121 20104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.4:42703 every 8 connection(s)
I20240501 19:55:57.762912 19977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:55:57.763338 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 19977
I20240501 19:55:57.763451 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:55:57.765552 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.5:0
--local_ip_for_outbound_sockets=127.17.65.5
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:57.771054 20105 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:55:57.771253 20105 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.771646 20105 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:55:57.772504 19524 ts_manager.cc:194] Registered new tserver with Master: 2d6860caa69249c1837378d6f5247099 (127.17.65.4:42703)
I20240501 19:55:57.773034 19524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.4:42917
W20240501 19:55:57.866024 20108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.866210 20108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.866278 20108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.869210 20108 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:57.869277 20108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.869362 20108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.5
I20240501 19:55:57.872586 20108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.20108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.5
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.873332 20108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.874321 20108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.882170 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.882766 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.883345 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.884307 20108 server_base.cc:1034] running on GCE node
I20240501 19:55:57.884581 20108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.884994 20108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.886121 20108 hybrid_clock.cc:648] HybridClock initialized: now 1714593357886110 us; error 24 us; skew 500 ppm
I20240501 19:55:57.887784 20108 webserver.cc:460] Webserver started at http://127.17.65.5:34769/ using document root <none> and password file <none>
I20240501 19:55:57.888231 20108 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.888280 20108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.888446 20108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.890005 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "fd3145d4045844a99067befa34a077be"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.890426 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "fd3145d4045844a99067befa34a077be"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.893493 20108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:57.895259 20122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.896181 20108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 19:55:57.896281 20108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fd3145d4045844a99067befa34a077be"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:55:57.896359 20108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.905157 20108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.905634 20108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.905789 20108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.906217 20108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.906986 20108 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.907054 20108 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.907114 20108 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.907138 20108 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.914753 20108 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.5:41697
I20240501 19:55:57.914773 20235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.5:41697 every 8 connection(s)
I20240501 19:55:57.915534 20108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:55:57.920351 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 20108
I20240501 19:55:57.920454 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:55:57.922382 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.6:0
--local_ip_for_outbound_sockets=127.17.65.6
--webserver_interface=127.17.65.6
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:57.923362 20236 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:55:57.923540 20236 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.924010 20236 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:55:57.924924 19524 ts_manager.cc:194] Registered new tserver with Master: fd3145d4045844a99067befa34a077be (127.17.65.5:41697)
I20240501 19:55:57.925673 19524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.5:43523
W20240501 19:55:58.021279 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.021456 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.021492 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.024127 20239 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:58.024196 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.024257 20239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.6
I20240501 19:55:58.027765 20239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.65.6
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.20239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.6
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.028613 20239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.029678 20239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.037923 20244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.038512 20247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.039364 20245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.040182 20239 server_base.cc:1034] running on GCE node
I20240501 19:55:58.040493 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.040984 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.042117 20239 hybrid_clock.cc:648] HybridClock initialized: now 1714593358042107 us; error 24 us; skew 500 ppm
I20240501 19:55:58.043782 20239 webserver.cc:460] Webserver started at http://127.17.65.6:43011/ using document root <none> and password file <none>
I20240501 19:55:58.044333 20239 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.044384 20239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.044544 20239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.046133 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "5bdf040be08b4aa292256860320503f7"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-703j"
I20240501 19:55:58.046582 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "5bdf040be08b4aa292256860320503f7"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-703j"
I20240501 19:55:58.049721 20239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:58.051829 20253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.052919 20239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.053020 20239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "5bdf040be08b4aa292256860320503f7"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-703j"
I20240501 19:55:58.053092 20239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.064903 20239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.065402 20239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.065534 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.066023 20239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.066951 20239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.066998 20239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.067109 20239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.067147 20239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.075208 20239 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.6:42633
I20240501 19:55:58.075245 20366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.6:42633 every 8 connection(s)
I20240501 19:55:58.076102 20239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:55:58.077893 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 20239
I20240501 19:55:58.078006 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:55:58.080065 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.7:0
--local_ip_for_outbound_sockets=127.17.65.7
--webserver_interface=127.17.65.7
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:55:58.084515 20367 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:55:58.084698 20367 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.085075 20367 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:55:58.085934 19524 ts_manager.cc:194] Registered new tserver with Master: 5bdf040be08b4aa292256860320503f7 (127.17.65.6:42633)
I20240501 19:55:58.086454 19524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.6:42963
W20240501 19:55:58.178999 20370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.179198 20370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.179239 20370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.182066 20370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:55:58.182121 20370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.182211 20370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.7
I20240501 19:55:58.185395 20370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.65.7
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.20370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.7
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.186149 20370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.187202 20370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.195091 20375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.195936 20378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.196635 20376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.197324 20370 server_base.cc:1034] running on GCE node
I20240501 19:55:58.197654 20370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.198110 20370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.199247 20370 hybrid_clock.cc:648] HybridClock initialized: now 1714593358199233 us; error 22 us; skew 500 ppm
I20240501 19:55:58.200798 20370 webserver.cc:460] Webserver started at http://127.17.65.7:44495/ using document root <none> and password file <none>
I20240501 19:55:58.201251 20370 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.201301 20370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.201434 20370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.202857 20370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "fa2c654d61eb4d32967da7ea2374d738"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-703j"
I20240501 19:55:58.203270 20370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "fa2c654d61eb4d32967da7ea2374d738"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-703j"
I20240501 19:55:58.206100 20370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:55:58.208020 20384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.208837 20370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:58.208926 20370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fa2c654d61eb4d32967da7ea2374d738"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-703j"
I20240501 19:55:58.208993 20370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.236804 20370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.237466 20370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.237612 20370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.238121 20370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.238956 20370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.239001 20370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.239070 20370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.239099 20370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.247839 20370 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.7:34015
I20240501 19:55:58.247876 20497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.7:34015 every 8 connection(s)
I20240501 19:55:58.248695 20370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:55:58.256315 20498 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:55:58.256335 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 20370
I20240501 19:55:58.256448 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:55:58.256497 20498 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.256886 20498 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:55:58.257727 19524 ts_manager.cc:194] Registered new tserver with Master: fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7:34015)
I20240501 19:55:58.258271 19524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.7:58029
I20240501 19:55:58.258473 17668 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 19:55:58.266621 19524 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41650:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:55:58.280795 19777 tablet_service.cc:1467] Processing CreateTablet for tablet ecdb17582af74a12b63acbbe2560166d (DEFAULT_TABLE table=TestTable [id=11ff0b9ce562494cad1b0f88aceb3660]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:58.281569 19777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecdb17582af74a12b63acbbe2560166d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:58.283489 19712 heartbeater.cc:503] Master 127.17.65.62:37761 was elected leader, sending a full tablet report...
I20240501 19:55:58.286201 20512 tablet_bootstrap.cc:492] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: Bootstrap starting.
I20240501 19:55:58.288025 20512 tablet_bootstrap.cc:654] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:58.288659 20512 log.cc:824] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:58.289966 20512 tablet_bootstrap.cc:492] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: No bootstrap required, opened a new log
I20240501 19:55:58.290117 20512 ts_tablet_manager.cc:1367] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:55:58.292320 20512 raft_consensus.cc:348] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:55:58.292464 20512 raft_consensus.cc:374] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:58.292520 20512 raft_consensus.cc:729] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Initialized, Role: FOLLOWER
I20240501 19:55:58.292850 20512 consensus_queue.cc:260] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:55:58.292953 20512 raft_consensus.cc:388] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:58.293006 20512 raft_consensus.cc:482] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:58.293094 20512 raft_consensus.cc:3037] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:58.294008 20512 raft_consensus.cc:504] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:55:58.294268 20512 leader_election.cc:302] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6; no voters:
I20240501 19:55:58.294520 20512 leader_election.cc:288] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:58.294602 20514 raft_consensus.cc:2781] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:58.294803 20514 raft_consensus.cc:686] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:55:58.295051 19843 heartbeater.cc:503] Master 127.17.65.62:37761 was elected leader, sending a full tablet report...
I20240501 19:55:58.295176 20514 consensus_queue.cc:237] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:55:58.295300 20512 ts_tablet_manager.cc:1398] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 19:55:58.297909 19524 catalog_manager.cc:5484] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } }
I20240501 19:55:58.305928 17668 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 19:55:58.307070 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 002bd5b7fe4c40169448f30e32ec54c6 to finish bootstrapping
I20240501 19:55:58.312078 17668 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1af1fefd33104e7bad976e2bd786c0b6 to finish bootstrapping
I20240501 19:55:58.315996 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b01a4a9860ce4bff90c3dc414228d44e to finish bootstrapping
I20240501 19:55:58.321094 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2d6860caa69249c1837378d6f5247099 to finish bootstrapping
I20240501 19:55:58.325680 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fd3145d4045844a99067befa34a077be to finish bootstrapping
I20240501 19:55:58.330776 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5bdf040be08b4aa292256860320503f7 to finish bootstrapping
I20240501 19:55:58.335502 17668 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fa2c654d61eb4d32967da7ea2374d738 to finish bootstrapping
I20240501 19:55:58.340129 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 19846
I20240501 19:55:58.344950 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 19977
I20240501 19:55:58.350136 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 20108
I20240501 19:55:58.354940 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 20239
I20240501 19:55:58.359645 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 20370
I20240501 19:56:02.115760 19524 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41650:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:02.127141 19645 tablet_service.cc:1467] Processing CreateTablet for tablet 089c2d27c8c2479e8efafa68ebf12969 (DEFAULT_TABLE table=rebalance_test_table_0 [id=aa25c285e4c541c6b76edca67986e9af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.127128 19644 tablet_service.cc:1467] Processing CreateTablet for tablet 678cb6a860ce43f3a52aab82fa7540a9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=aa25c285e4c541c6b76edca67986e9af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.127128 19646 tablet_service.cc:1467] Processing CreateTablet for tablet 97866b7451304498ab8e4300f6853e7b (DEFAULT_TABLE table=rebalance_test_table_0 [id=aa25c285e4c541c6b76edca67986e9af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.128180 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089c2d27c8c2479e8efafa68ebf12969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.128475 19646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97866b7451304498ab8e4300f6853e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.128602 19644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678cb6a860ce43f3a52aab82fa7540a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.133181 20532 tablet_bootstrap.cc:492] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:02.134860 20532 tablet_bootstrap.cc:654] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.135684 20532 log.cc:824] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:02.137017 20532 tablet_bootstrap.cc:492] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:02.137166 20532 ts_tablet_manager.cc:1367] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:02.139372 20532 raft_consensus.cc:348] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.139534 20532 raft_consensus.cc:374] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.139597 20532 raft_consensus.cc:729] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.139925 20532 consensus_queue.cc:260] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.140026 20532 raft_consensus.cc:388] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.140084 20532 raft_consensus.cc:482] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.140177 20532 raft_consensus.cc:3037] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.140781 20532 raft_consensus.cc:504] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.141047 20532 leader_election.cc:302] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:02.141270 20532 leader_election.cc:288] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.141389 20534 raft_consensus.cc:2781] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.141609 20534 raft_consensus.cc:686] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:02.141928 20534 consensus_queue.cc:237] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.142197 20532 ts_tablet_manager.cc:1398] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:02.142381 20532 tablet_bootstrap.cc:492] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:02.143157 20532 tablet_bootstrap.cc:654] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.144246 20532 tablet_bootstrap.cc:492] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:02.144322 20532 ts_tablet_manager.cc:1367] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:02.144434 19524 catalog_manager.cc:5484] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
I20240501 19:56:02.144801 20532 raft_consensus.cc:348] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.144915 20532 raft_consensus.cc:374] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.144964 20532 raft_consensus.cc:729] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.145068 20532 consensus_queue.cc:260] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.145138 20532 raft_consensus.cc:388] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.145159 20532 raft_consensus.cc:482] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.145197 20532 raft_consensus.cc:3037] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.145807 20532 raft_consensus.cc:504] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.145923 20532 leader_election.cc:302] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:02.146029 20532 leader_election.cc:288] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.146238 20532 ts_tablet_manager.cc:1398] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:02.146392 20534 raft_consensus.cc:2781] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.146394 20532 tablet_bootstrap.cc:492] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:02.146492 20534 raft_consensus.cc:686] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:02.146600 20534 consensus_queue.cc:237] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.147435 20532 tablet_bootstrap.cc:654] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.147789 19524 catalog_manager.cc:5484] T 97866b7451304498ab8e4300f6853e7b P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
I20240501 19:56:02.148372 20532 tablet_bootstrap.cc:492] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:02.148442 20532 ts_tablet_manager.cc:1367] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:02.148936 20532 raft_consensus.cc:348] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.149006 20532 raft_consensus.cc:374] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.149039 20532 raft_consensus.cc:729] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.149171 20532 consensus_queue.cc:260] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.149242 20532 raft_consensus.cc:388] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.149264 20532 raft_consensus.cc:482] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.149292 20532 raft_consensus.cc:3037] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.149919 20532 raft_consensus.cc:504] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.150028 20532 leader_election.cc:302] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:02.150112 20532 leader_election.cc:288] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.150166 20535 raft_consensus.cc:2781] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.150257 20535 raft_consensus.cc:686] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:02.150354 20535 consensus_queue.cc:237] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.150413 20532 ts_tablet_manager.cc:1398] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.151681 19523 catalog_manager.cc:5484] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
W20240501 19:56:02.259721 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:02.259955 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:02.262898 20543 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 79.9976 ms
time per row: 0.0399988 ms
I20240501 19:56:02.382599 19517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41650:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:02.388193 19776 tablet_service.cc:1467] Processing CreateTablet for tablet afb520db25524bc0b8deee1f443f6a44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06418a4164d9480faaf291fc1c1d0a69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.388193 19775 tablet_service.cc:1467] Processing CreateTablet for tablet e05bb1fdf29d48b99d3aa029e315a519 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06418a4164d9480faaf291fc1c1d0a69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.388195 19777 tablet_service.cc:1467] Processing CreateTablet for tablet 8c8e23f52a6e46aeae2d71e1ed04e9a7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06418a4164d9480faaf291fc1c1d0a69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.388577 19776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb520db25524bc0b8deee1f443f6a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.388727 19775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05bb1fdf29d48b99d3aa029e315a519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.390410 19777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c8e23f52a6e46aeae2d71e1ed04e9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.391129 20575 tablet_bootstrap.cc:492] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: Bootstrap starting.
I20240501 19:56:02.392112 20575 tablet_bootstrap.cc:654] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.393123 20575 tablet_bootstrap.cc:492] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: No bootstrap required, opened a new log
I20240501 19:56:02.393204 20575 ts_tablet_manager.cc:1367] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.393783 20575 raft_consensus.cc:348] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.393872 20575 raft_consensus.cc:374] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.393908 20575 raft_consensus.cc:729] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.394032 20575 consensus_queue.cc:260] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.394099 20575 raft_consensus.cc:388] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.394145 20575 raft_consensus.cc:482] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.394194 20575 raft_consensus.cc:3037] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.394837 20575 raft_consensus.cc:504] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.394963 20575 leader_election.cc:302] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6; no voters:
I20240501 19:56:02.395131 20575 leader_election.cc:288] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.395234 20577 raft_consensus.cc:2781] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.395421 20575 ts_tablet_manager.cc:1398] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.395438 20577 raft_consensus.cc:686] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:02.395614 20575 tablet_bootstrap.cc:492] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: Bootstrap starting.
I20240501 19:56:02.395583 20577 consensus_queue.cc:237] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.396467 20575 tablet_bootstrap.cc:654] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.396983 19517 catalog_manager.cc:5484] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } }
I20240501 19:56:02.397579 20575 tablet_bootstrap.cc:492] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: No bootstrap required, opened a new log
I20240501 19:56:02.397660 20575 ts_tablet_manager.cc:1367] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:02.398061 20575 raft_consensus.cc:348] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.398149 20575 raft_consensus.cc:374] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.398185 20575 raft_consensus.cc:729] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.398294 20575 consensus_queue.cc:260] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.398363 20575 raft_consensus.cc:388] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.398391 20575 raft_consensus.cc:482] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.398429 20575 raft_consensus.cc:3037] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.399186 20575 raft_consensus.cc:504] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.399317 20575 leader_election.cc:302] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6; no voters:
I20240501 19:56:02.399403 20575 leader_election.cc:288] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.399590 20575 ts_tablet_manager.cc:1398] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:02.399575 20577 raft_consensus.cc:2781] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.399785 20575 tablet_bootstrap.cc:492] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6: Bootstrap starting.
I20240501 19:56:02.399847 20577 raft_consensus.cc:686] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:02.399955 20577 consensus_queue.cc:237] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.400949 20575 tablet_bootstrap.cc:654] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.401461 19517 catalog_manager.cc:5484] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } }
I20240501 19:56:02.402052 20575 tablet_bootstrap.cc:492] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6: No bootstrap required, opened a new log
I20240501 19:56:02.402127 20575 ts_tablet_manager.cc:1367] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:02.402468 20575 raft_consensus.cc:348] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.402539 20575 raft_consensus.cc:374] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.402570 20575 raft_consensus.cc:729] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.402711 20575 consensus_queue.cc:260] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.402801 20575 raft_consensus.cc:388] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.402841 20575 raft_consensus.cc:482] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.402890 20575 raft_consensus.cc:3037] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.403734 20575 raft_consensus.cc:504] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.403834 20575 leader_election.cc:302] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6; no voters:
I20240501 19:56:02.403899 20575 leader_election.cc:288] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.403959 20577 raft_consensus.cc:2781] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.404049 20577 raft_consensus.cc:686] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:02.404096 20575 ts_tablet_manager.cc:1398] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:02.404141 20577 consensus_queue.cc:237] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.405186 19517 catalog_manager.cc:5484] T 8c8e23f52a6e46aeae2d71e1ed04e9a7 P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } }
W20240501 19:56:02.519152 20586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:02.519419 20586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:02.522424 20586 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 68.0955 ms
time per row: 0.0340477 ms
I20240501 19:56:02.628939 19515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41650:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:02.633955 19644 tablet_service.cc:1467] Processing CreateTablet for tablet 251b0a3550a44aa7b266d253b479a5b6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a641b337afe4629915afc6453555f8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.633996 19646 tablet_service.cc:1467] Processing CreateTablet for tablet 68a0476d7a0c4d1f8dda6182816f4a17 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a641b337afe4629915afc6453555f8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.634317 19644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251b0a3550a44aa7b266d253b479a5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.634615 19646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a0476d7a0c4d1f8dda6182816f4a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.633955 19645 tablet_service.cc:1467] Processing CreateTablet for tablet 2de3fc3f434240748a8f75260808f9e5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a641b337afe4629915afc6453555f8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.635249 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de3fc3f434240748a8f75260808f9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.636196 20532 tablet_bootstrap.cc:492] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:02.637115 20532 tablet_bootstrap.cc:654] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.638092 20532 tablet_bootstrap.cc:492] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:02.638168 20532 ts_tablet_manager.cc:1367] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:02.638638 20532 raft_consensus.cc:348] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.638727 20532 raft_consensus.cc:374] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.638760 20532 raft_consensus.cc:729] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.638900 20532 consensus_queue.cc:260] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.638965 20532 raft_consensus.cc:388] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.639008 20532 raft_consensus.cc:482] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.639082 20532 raft_consensus.cc:3037] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.639811 20532 raft_consensus.cc:504] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.639956 20532 leader_election.cc:302] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:02.640055 20532 leader_election.cc:288] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.640125 20536 raft_consensus.cc:2781] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.640211 20536 raft_consensus.cc:686] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:02.640280 20532 ts_tablet_manager.cc:1398] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.640322 20536 consensus_queue.cc:237] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.640399 20532 tablet_bootstrap.cc:492] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:02.641181 20532 tablet_bootstrap.cc:654] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.641932 19515 catalog_manager.cc:5484] T 251b0a3550a44aa7b266d253b479a5b6 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
I20240501 19:56:02.642036 20532 tablet_bootstrap.cc:492] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:02.642108 20532 ts_tablet_manager.cc:1367] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:02.642467 20532 raft_consensus.cc:348] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.642561 20532 raft_consensus.cc:374] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.642591 20532 raft_consensus.cc:729] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.642697 20532 consensus_queue.cc:260] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.642776 20532 raft_consensus.cc:388] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.642849 20532 raft_consensus.cc:482] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.642904 20532 raft_consensus.cc:3037] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.644059 20532 raft_consensus.cc:504] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.644181 20532 leader_election.cc:302] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:02.644285 20532 leader_election.cc:288] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.644320 20536 raft_consensus.cc:2781] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.644392 20536 raft_consensus.cc:686] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:02.644490 20536 consensus_queue.cc:237] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.644567 20532 ts_tablet_manager.cc:1398] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.644690 20532 tablet_bootstrap.cc:492] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:02.645411 20532 tablet_bootstrap.cc:654] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.646018 19515 catalog_manager.cc:5484] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
I20240501 19:56:02.646325 20532 tablet_bootstrap.cc:492] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:02.646436 20532 ts_tablet_manager.cc:1367] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:02.646843 20532 raft_consensus.cc:348] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.646931 20532 raft_consensus.cc:374] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.646952 20532 raft_consensus.cc:729] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.647083 20532 consensus_queue.cc:260] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.647171 20532 raft_consensus.cc:388] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.647253 20532 raft_consensus.cc:482] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.647306 20532 raft_consensus.cc:3037] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.647976 20532 raft_consensus.cc:504] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.648072 20532 leader_election.cc:302] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:02.648136 20532 leader_election.cc:288] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.648212 20536 raft_consensus.cc:2781] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.648290 20536 raft_consensus.cc:686] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:02.648344 20532 ts_tablet_manager.cc:1398] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:02.648383 20536 consensus_queue.cc:237] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:02.649369 19515 catalog_manager.cc:5484] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
W20240501 19:56:02.752424 20624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:02.752658 20624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:02.755613 20624 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 63.0558 ms
time per row: 0.0315279 ms
I20240501 19:56:02.857944 19516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41650:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:02.863314 19777 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec069697dd9474fab37c77aa8abdb28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6e9d5ba6b7a740b49a263f77826f6763]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.863314 19775 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3b72769a4c41349d9052c8e740f009 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6e9d5ba6b7a740b49a263f77826f6763]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.863315 19776 tablet_service.cc:1467] Processing CreateTablet for tablet 32c3ffbbb4d140bbae02fdb2fed1e109 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6e9d5ba6b7a740b49a263f77826f6763]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.863651 19777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec069697dd9474fab37c77aa8abdb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.863802 19775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3b72769a4c41349d9052c8e740f009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.864040 19776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c3ffbbb4d140bbae02fdb2fed1e109. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.865559 20575 tablet_bootstrap.cc:492] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: Bootstrap starting.
I20240501 19:56:02.866333 20575 tablet_bootstrap.cc:654] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.867270 20575 tablet_bootstrap.cc:492] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: No bootstrap required, opened a new log
I20240501 19:56:02.867352 20575 ts_tablet_manager.cc:1367] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:02.867681 20575 raft_consensus.cc:348] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.867761 20575 raft_consensus.cc:374] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.867796 20575 raft_consensus.cc:729] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.867916 20575 consensus_queue.cc:260] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.867978 20575 raft_consensus.cc:388] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.868000 20575 raft_consensus.cc:482] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.868028 20575 raft_consensus.cc:3037] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.868599 20575 raft_consensus.cc:504] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.868729 20575 leader_election.cc:302] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6; no voters:
I20240501 19:56:02.868837 20575 leader_election.cc:288] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.868897 20579 raft_consensus.cc:2781] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.868988 20579 raft_consensus.cc:686] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:02.869100 20579 consensus_queue.cc:237] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.869372 20575 ts_tablet_manager.cc:1398] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.870204 20575 tablet_bootstrap.cc:492] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6: Bootstrap starting.
I20240501 19:56:02.870898 20575 tablet_bootstrap.cc:654] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.870868 19516 catalog_manager.cc:5484] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } }
I20240501 19:56:02.871896 20575 tablet_bootstrap.cc:492] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6: No bootstrap required, opened a new log
I20240501 19:56:02.871974 20575 ts_tablet_manager.cc:1367] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:02.872375 20575 raft_consensus.cc:348] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.872521 20575 raft_consensus.cc:374] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.872596 20575 raft_consensus.cc:729] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.872746 20575 consensus_queue.cc:260] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.872833 20575 raft_consensus.cc:388] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.872855 20575 raft_consensus.cc:482] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.872902 20575 raft_consensus.cc:3037] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.873605 20575 raft_consensus.cc:504] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.873726 20575 leader_election.cc:302] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6; no voters:
I20240501 19:56:02.873805 20575 leader_election.cc:288] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.873862 20578 raft_consensus.cc:2781] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.873946 20578 raft_consensus.cc:686] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:02.874044 20575 ts_tablet_manager.cc:1398] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.874045 20578 consensus_queue.cc:237] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.874249 20575 tablet_bootstrap.cc:492] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: Bootstrap starting.
I20240501 19:56:02.874969 20575 tablet_bootstrap.cc:654] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.875419 19516 catalog_manager.cc:5484] T 6a3b72769a4c41349d9052c8e740f009 P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } }
I20240501 19:56:02.876621 20575 tablet_bootstrap.cc:492] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: No bootstrap required, opened a new log
I20240501 19:56:02.876713 20575 ts_tablet_manager.cc:1367] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.877133 20575 raft_consensus.cc:348] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.877219 20575 raft_consensus.cc:374] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.877259 20575 raft_consensus.cc:729] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.877382 20575 consensus_queue.cc:260] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.877454 20575 raft_consensus.cc:388] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:02.877498 20575 raft_consensus.cc:482] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:02.877544 20575 raft_consensus.cc:3037] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.878136 20575 raft_consensus.cc:504] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.878243 20575 leader_election.cc:302] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6; no voters:
I20240501 19:56:02.878340 20575 leader_election.cc:288] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:02.878422 20578 raft_consensus.cc:2781] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.878523 20578 raft_consensus.cc:686] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:02.878609 20575 ts_tablet_manager.cc:1398] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.878641 20578 consensus_queue.cc:237] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:02.879756 19516 catalog_manager.cc:5484] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } }
W20240501 19:56:02.989590 20654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:02.989871 20654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:02.992964 20654 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 68.5411 ms
time per row: 0.0342706 ms
I20240501 19:56:03.100976 19516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41650:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:03.106328 19646 tablet_service.cc:1467] Processing CreateTablet for tablet c57a563e77234a7b99dc890e2d5b2fa9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=eec65c7f9008472d9417285bc1f2057b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.106365 19644 tablet_service.cc:1467] Processing CreateTablet for tablet 618965d7b3d44bf58c00664087392002 (DEFAULT_TABLE table=rebalance_test_table_4 [id=eec65c7f9008472d9417285bc1f2057b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.106709 19646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57a563e77234a7b99dc890e2d5b2fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.106842 19644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618965d7b3d44bf58c00664087392002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.106328 19645 tablet_service.cc:1467] Processing CreateTablet for tablet 6fae90ed925f41999fe254adcf950841 (DEFAULT_TABLE table=rebalance_test_table_4 [id=eec65c7f9008472d9417285bc1f2057b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.108350 20532 tablet_bootstrap.cc:492] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:03.108464 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fae90ed925f41999fe254adcf950841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.109216 20532 tablet_bootstrap.cc:654] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.110144 20532 tablet_bootstrap.cc:492] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:03.110214 20532 ts_tablet_manager.cc:1367] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:03.110579 20532 raft_consensus.cc:348] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.110661 20532 raft_consensus.cc:374] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.110693 20532 raft_consensus.cc:729] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.110821 20532 consensus_queue.cc:260] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.110894 20532 raft_consensus.cc:388] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:03.110937 20532 raft_consensus.cc:482] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:03.110991 20532 raft_consensus.cc:3037] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.111640 20532 raft_consensus.cc:504] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.111750 20532 leader_election.cc:302] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:03.111846 20532 leader_election.cc:288] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:03.111887 20536 raft_consensus.cc:2781] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.111974 20536 raft_consensus.cc:686] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:03.112067 20532 ts_tablet_manager.cc:1398] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:03.112087 20536 consensus_queue.cc:237] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.112179 20532 tablet_bootstrap.cc:492] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:03.112979 20532 tablet_bootstrap.cc:654] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.113287 19516 catalog_manager.cc:5484] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
I20240501 19:56:03.113992 20532 tablet_bootstrap.cc:492] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:03.114073 20532 ts_tablet_manager.cc:1367] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:03.114434 20532 raft_consensus.cc:348] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.114521 20532 raft_consensus.cc:374] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.114559 20532 raft_consensus.cc:729] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.114688 20532 consensus_queue.cc:260] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.114774 20532 raft_consensus.cc:388] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:03.114820 20532 raft_consensus.cc:482] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:03.114867 20532 raft_consensus.cc:3037] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.115509 20532 raft_consensus.cc:504] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.115608 20532 leader_election.cc:302] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:03.115684 20532 leader_election.cc:288] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:03.115733 20536 raft_consensus.cc:2781] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.115830 20536 raft_consensus.cc:686] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:03.115952 20532 ts_tablet_manager.cc:1398] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:03.115932 20536 consensus_queue.cc:237] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.116097 20532 tablet_bootstrap.cc:492] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: Bootstrap starting.
I20240501 19:56:03.116938 20532 tablet_bootstrap.cc:654] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.117678 19516 catalog_manager.cc:5484] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
I20240501 19:56:03.117924 20532 tablet_bootstrap.cc:492] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: No bootstrap required, opened a new log
I20240501 19:56:03.117997 20532 ts_tablet_manager.cc:1367] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:03.118369 20532 raft_consensus.cc:348] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.118460 20532 raft_consensus.cc:374] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.118503 20532 raft_consensus.cc:729] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.118619 20532 consensus_queue.cc:260] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.118686 20532 raft_consensus.cc:388] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:03.118721 20532 raft_consensus.cc:482] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:03.118764 20532 raft_consensus.cc:3037] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.119503 20532 raft_consensus.cc:504] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.119617 20532 leader_election.cc:302] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6; no voters:
I20240501 19:56:03.119715 20532 leader_election.cc:288] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:03.119766 20536 raft_consensus.cc:2781] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.119899 20536 raft_consensus.cc:686] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:03.119941 20532 ts_tablet_manager.cc:1398] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:03.120018 20536 consensus_queue.cc:237] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:03.121160 19516 catalog_manager.cc:5484] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } }
W20240501 19:56:03.228217 20684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:03.228430 20684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:03.231189 20684 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 80.3925 ms
time per row: 0.0401963 ms
I20240501 19:56:03.362910 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.3:44125
--local_ip_for_outbound_sockets=127.17.65.3
--tserver_master_addrs=127.17.65.62:37761
--webserver_port=39217
--webserver_interface=127.17.65.3
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:56:03.472177 20708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:03.472411 20708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:03.472472 20708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:03.475677 20708 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:03.475757 20708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:03.475862 20708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.3
I20240501 19:56:03.479601 20708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.3:44125
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.65.3
--webserver_port=39217
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.20708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.3
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:03.480479 20708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:03.481591 20708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:03.490536 20714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:03.491184 20708 server_base.cc:1034] running on GCE node
W20240501 19:56:03.492023 20713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:03.492787 20716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:03.493153 20708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:03.493721 20708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:03.494881 20708 hybrid_clock.cc:648] HybridClock initialized: now 1714593363494872 us; error 30 us; skew 500 ppm
I20240501 19:56:03.496711 20708 webserver.cc:460] Webserver started at http://127.17.65.3:39217/ using document root <none> and password file <none>
I20240501 19:56:03.497253 20708 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:03.497332 20708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:03.500665 20708 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:03.502482 20722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.503520 20708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:03.503623 20708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b01a4a9860ce4bff90c3dc414228d44e"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:56:03.504024 20708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:03.540284 20708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:03.540962 20708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:03.541126 20708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:03.541703 20708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:03.542624 20708 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:03.542670 20708 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.542713 20708 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:03.542742 20708 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.551265 20708 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.3:44125
I20240501 19:56:03.551287 20835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.3:44125 every 8 connection(s)
I20240501 19:56:03.552242 20708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:03.560138 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 20708
I20240501 19:56:03.560632 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.4:42703
--local_ip_for_outbound_sockets=127.17.65.4
--tserver_master_addrs=127.17.65.62:37761
--webserver_port=37191
--webserver_interface=127.17.65.4
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:03.560865 20836 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:56:03.561091 20836 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:03.561522 20836 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:56:03.562669 19516 ts_manager.cc:194] Re-registered known tserver with Master: b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3:44125)
I20240501 19:56:03.563257 19516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.3:36549
W20240501 19:56:03.666541 20839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:03.666746 20839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:03.666787 20839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:03.669765 20839 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:03.669844 20839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:03.669929 20839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.4
I20240501 19:56:03.673892 20839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.4:42703
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.65.4
--webserver_port=37191
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.20839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.4
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:03.674786 20839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:03.675935 20839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:03.684198 20844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:03.684764 20845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:03.685721 20847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:03.686509 20839 server_base.cc:1034] running on GCE node
I20240501 19:56:03.686849 20839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:03.687403 20839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:03.688548 20839 hybrid_clock.cc:648] HybridClock initialized: now 1714593363688525 us; error 40 us; skew 500 ppm
I20240501 19:56:03.690213 20839 webserver.cc:460] Webserver started at http://127.17.65.4:37191/ using document root <none> and password file <none>
I20240501 19:56:03.690747 20839 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:03.690824 20839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:03.694077 20839 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:03.695907 20853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.696915 20839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:03.697003 20839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2d6860caa69249c1837378d6f5247099"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:56:03.697376 20839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:03.709509 20839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:03.710045 20839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:03.710201 20839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:03.710662 20839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:03.711493 20839 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:03.711537 20839 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.711581 20839 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:03.711602 20839 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.719547 20839 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.4:42703
I20240501 19:56:03.719580 20966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.4:42703 every 8 connection(s)
I20240501 19:56:03.720485 20839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:03.726724 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 20839
I20240501 19:56:03.727167 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.5:41697
--local_ip_for_outbound_sockets=127.17.65.5
--tserver_master_addrs=127.17.65.62:37761
--webserver_port=34769
--webserver_interface=127.17.65.5
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:03.729153 20967 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:56:03.729363 20967 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:03.729851 20967 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:56:03.730837 19516 ts_manager.cc:194] Re-registered known tserver with Master: 2d6860caa69249c1837378d6f5247099 (127.17.65.4:42703)
I20240501 19:56:03.731386 19516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.4:52171
W20240501 19:56:03.830507 20970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:03.830689 20970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:03.830753 20970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:03.833626 20970 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:03.833690 20970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:03.833775 20970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.5
I20240501 19:56:03.837368 20970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.5:41697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.65.5
--webserver_port=34769
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.20970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.5
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:03.838176 20970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:03.839260 20970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:03.847220 20975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:03.847805 20976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:03.848780 20978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:03.849541 20970 server_base.cc:1034] running on GCE node
I20240501 19:56:03.849853 20970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:03.850318 20970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:03.851465 20970 hybrid_clock.cc:648] HybridClock initialized: now 1714593363851456 us; error 23 us; skew 500 ppm
I20240501 19:56:03.853034 20970 webserver.cc:460] Webserver started at http://127.17.65.5:34769/ using document root <none> and password file <none>
I20240501 19:56:03.853525 20970 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:03.853588 20970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:03.856638 20970 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:03.858315 20984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.859323 20970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:03.859433 20970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fd3145d4045844a99067befa34a077be"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-703j"
I20240501 19:56:03.859823 20970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:03.881184 20970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:03.881816 20970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:03.881953 20970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:03.882536 20970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:03.883472 20970 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:03.883530 20970 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.883586 20970 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:03.883611 20970 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:03.891443 20970 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.5:41697
I20240501 19:56:03.891482 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.5:41697 every 8 connection(s)
I20240501 19:56:03.892356 20970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:03.893110 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 20970
I20240501 19:56:03.893421 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.6:42633
--local_ip_for_outbound_sockets=127.17.65.6
--tserver_master_addrs=127.17.65.62:37761
--webserver_port=43011
--webserver_interface=127.17.65.6
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:03.901428 21098 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:56:03.901620 21098 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:03.902009 21098 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:56:03.902828 19516 ts_manager.cc:194] Re-registered known tserver with Master: fd3145d4045844a99067befa34a077be (127.17.65.5:41697)
I20240501 19:56:03.903327 19516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.5:36689
W20240501 19:56:03.993793 21100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:03.994009 21100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:03.994071 21100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:03.996918 21100 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:03.996980 21100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:03.997081 21100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.6
I20240501 19:56:04.000367 21100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.6:42633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.65.6
--webserver_port=43011
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.21100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.6
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:04.001236 21100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:04.002368 21100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:04.010262 21107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.010847 21109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.011461 21106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:04.012454 21100 server_base.cc:1034] running on GCE node
I20240501 19:56:04.012780 21100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:04.013232 21100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:04.014387 21100 hybrid_clock.cc:648] HybridClock initialized: now 1714593364014368 us; error 36 us; skew 500 ppm
I20240501 19:56:04.015962 21100 webserver.cc:460] Webserver started at http://127.17.65.6:43011/ using document root <none> and password file <none>
I20240501 19:56:04.016502 21100 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:04.016574 21100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:04.019627 21100 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:04.021304 21115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.022336 21100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:04.022439 21100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "5bdf040be08b4aa292256860320503f7"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-703j"
I20240501 19:56:04.022831 21100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:04.038892 21100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:04.039490 21100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:04.039634 21100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:04.040112 21100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:04.040930 21100 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:04.040977 21100 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.041028 21100 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:04.041062 21100 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.049134 21100 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.6:42633
I20240501 19:56:04.049155 21228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.6:42633 every 8 connection(s)
I20240501 19:56:04.050137 21100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:04.058079 21229 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:56:04.058285 21229 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:04.058718 21229 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:56:04.058734 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 21100
I20240501 19:56:04.059113 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.7:34015
--local_ip_for_outbound_sockets=127.17.65.7
--tserver_master_addrs=127.17.65.62:37761
--webserver_port=44495
--webserver_interface=127.17.65.7
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:04.059805 19516 ts_manager.cc:194] Re-registered known tserver with Master: 5bdf040be08b4aa292256860320503f7 (127.17.65.6:42633)
I20240501 19:56:04.060525 19516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.6:60537
W20240501 19:56:04.163079 21232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.163306 21232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:04.163389 21232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.166260 21232 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:04.166337 21232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:04.166453 21232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.7
I20240501 19:56:04.169905 21232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.7:34015
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.65.7
--webserver_port=44495
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.21232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.7
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:04.170703 21232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:04.171742 21232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:04.179801 21238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.180368 21240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.181155 21237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:04.182005 21232 server_base.cc:1034] running on GCE node
I20240501 19:56:04.182325 21232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:04.182783 21232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:04.183917 21232 hybrid_clock.cc:648] HybridClock initialized: now 1714593364183904 us; error 32 us; skew 500 ppm
I20240501 19:56:04.185631 21232 webserver.cc:460] Webserver started at http://127.17.65.7:44495/ using document root <none> and password file <none>
I20240501 19:56:04.186156 21232 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:04.186230 21232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:04.189347 21232 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.190938 21246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.191834 21232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:04.191924 21232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fa2c654d61eb4d32967da7ea2374d738"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-703j"
I20240501 19:56:04.192303 21232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:04.212152 21232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:04.212687 21232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:04.212824 21232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:04.213268 21232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:04.214020 21232 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:04.214064 21232 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.214114 21232 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:04.214144 21232 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.221931 21232 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.7:34015
I20240501 19:56:04.221956 21359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.7:34015 every 8 connection(s)
I20240501 19:56:04.222851 21232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:04.224836 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 21232
I20240501 19:56:04.231400 21360 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:56:04.231616 21360 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:04.232148 21360 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:56:04.233081 19516 ts_manager.cc:194] Re-registered known tserver with Master: fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7:34015)
I20240501 19:56:04.233605 19516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.7:47305
I20240501 19:56:04.406706 19777 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:04.413599 21163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.415243 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.415877 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:04.417701 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.434839 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.443323 20770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.543104 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.543288 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:04.547245 21163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.548018 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.559438 20770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.565219 19777 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:04.571084 20836 heartbeater.cc:503] Master 127.17.65.62:37761 was elected leader, sending a full tablet report...
I20240501 19:56:04.579620 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:04.614758 19797 consensus_queue.cc:237] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
W20240501 19:56:04.619565 19732 consensus_peers.cc:485] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 -> Peer fd3145d4045844a99067befa34a077be (127.17.65.5:41697): Couldn't send request to peer fd3145d4045844a99067befa34a077be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ec069697dd9474fab37c77aa8abdb28. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:04.733597 20967 heartbeater.cc:503] Master 127.17.65.62:37761 was elected leader, sending a full tablet report...
I20240501 19:56:04.904865 21098 heartbeater.cc:503] Master 127.17.65.62:37761 was elected leader, sending a full tablet report...
I20240501 19:56:05.062657 21229 heartbeater.cc:503] Master 127.17.65.62:37761 was elected leader, sending a full tablet report...
I20240501 19:56:05.065706 21462 ts_tablet_manager.cc:902] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: Initiating tablet copy from peer 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:05.066452 21462 tablet_copy_client.cc:308] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:42021
I20240501 19:56:05.069622 19817 tablet_copy_service.cc:140] P 1af1fefd33104e7bad976e2bd786c0b6: Received BeginTabletCopySession request for tablet 8ec069697dd9474fab37c77aa8abdb28 from peer fd3145d4045844a99067befa34a077be ({username='slave'} at 127.17.65.5:37267)
I20240501 19:56:05.069743 19817 tablet_copy_service.cc:161] P 1af1fefd33104e7bad976e2bd786c0b6: Beginning new tablet copy session on tablet 8ec069697dd9474fab37c77aa8abdb28 from peer fd3145d4045844a99067befa34a077be at {username='slave'} at 127.17.65.5:37267: session id = fd3145d4045844a99067befa34a077be-8ec069697dd9474fab37c77aa8abdb28
I20240501 19:56:05.070858 19817 tablet_copy_source_session.cc:205] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:05.073866 21462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec069697dd9474fab37c77aa8abdb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.078740 21462 tablet_copy_client.cc:783] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:05.078902 21462 tablet_copy_client.cc:647] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:05.080425 21462 tablet_copy_client.cc:523] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:05.081666 21462 tablet_bootstrap.cc:492] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: Bootstrap starting.
I20240501 19:56:05.086516 21462 log.cc:824] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:05.105513 21462 tablet_bootstrap.cc:492] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:05.105911 21462 tablet_bootstrap.cc:492] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: Bootstrap complete.
I20240501 19:56:05.106184 21462 ts_tablet_manager.cc:1367] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240501 19:56:05.108354 21462 raft_consensus.cc:348] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:05.108911 21462 raft_consensus.cc:729] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3145d4045844a99067befa34a077be, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.109186 21462 consensus_queue.cc:260] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:05.109925 21462 ts_tablet_manager.cc:1398] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240501 19:56:05.110751 19817 tablet_copy_service.cc:342] P 1af1fefd33104e7bad976e2bd786c0b6: Request end of tablet copy session fd3145d4045844a99067befa34a077be-8ec069697dd9474fab37c77aa8abdb28 received from {username='slave'} at 127.17.65.5:37267
I20240501 19:56:05.110858 19817 tablet_copy_service.cc:433] P 1af1fefd33104e7bad976e2bd786c0b6: ending tablet copy session fd3145d4045844a99067befa34a077be-8ec069697dd9474fab37c77aa8abdb28 on tablet 8ec069697dd9474fab37c77aa8abdb28 with peer fd3145d4045844a99067befa34a077be
I20240501 19:56:05.235695 21360 heartbeater.cc:503] Master 127.17.65.62:37761 was elected leader, sending a full tablet report...
I20240501 19:56:05.563650 21052 raft_consensus.cc:1203] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:05.564996 21470 raft_consensus.cc:2932] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd3145d4045844a99067befa34a077be (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:05.565601 21052 raft_consensus.cc:2932] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd3145d4045844a99067befa34a077be (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:05.568115 19517 catalog_manager.cc:5484] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: config changed from index -1 to 4, VOTER fd3145d4045844a99067befa34a077be (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:05.585821 19666 consensus_queue.cc:237] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
W20240501 19:56:05.590668 19601 consensus_peers.cc:485] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 -> Peer fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7:34015): Couldn't send request to peer fa2c654d61eb4d32967da7ea2374d738. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 618965d7b3d44bf58c00664087392002. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:06.055526 21497 ts_tablet_manager.cc:902] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: Initiating tablet copy from peer 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:06.056231 21497 tablet_copy_client.cc:308] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:36451
I20240501 19:56:06.059830 19686 tablet_copy_service.cc:140] P 002bd5b7fe4c40169448f30e32ec54c6: Received BeginTabletCopySession request for tablet 618965d7b3d44bf58c00664087392002 from peer fa2c654d61eb4d32967da7ea2374d738 ({username='slave'} at 127.17.65.7:34891)
I20240501 19:56:06.059952 19686 tablet_copy_service.cc:161] P 002bd5b7fe4c40169448f30e32ec54c6: Beginning new tablet copy session on tablet 618965d7b3d44bf58c00664087392002 from peer fa2c654d61eb4d32967da7ea2374d738 at {username='slave'} at 127.17.65.7:34891: session id = fa2c654d61eb4d32967da7ea2374d738-618965d7b3d44bf58c00664087392002
I20240501 19:56:06.061151 19686 tablet_copy_source_session.cc:205] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:06.064226 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618965d7b3d44bf58c00664087392002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.069815 21497 tablet_copy_client.cc:783] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:06.070108 21497 tablet_copy_client.cc:647] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:06.071796 21497 tablet_copy_client.cc:523] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:06.073153 21497 tablet_bootstrap.cc:492] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: Bootstrap starting.
I20240501 19:56:06.078286 21497 log.cc:824] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:06.097038 21497 tablet_bootstrap.cc:492] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:06.097385 21497 tablet_bootstrap.cc:492] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: Bootstrap complete.
I20240501 19:56:06.097681 21497 ts_tablet_manager.cc:1367] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.005s
I20240501 19:56:06.099587 21497 raft_consensus.cc:348] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:06.100121 21497 raft_consensus.cc:729] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2c654d61eb4d32967da7ea2374d738, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.100389 21497 consensus_queue.cc:260] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:06.101147 21497 ts_tablet_manager.cc:1398] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 19:56:06.101871 19686 tablet_copy_service.cc:342] P 002bd5b7fe4c40169448f30e32ec54c6: Request end of tablet copy session fa2c654d61eb4d32967da7ea2374d738-618965d7b3d44bf58c00664087392002 received from {username='slave'} at 127.17.65.7:34891
I20240501 19:56:06.101958 19686 tablet_copy_service.cc:433] P 002bd5b7fe4c40169448f30e32ec54c6: ending tablet copy session fa2c654d61eb4d32967da7ea2374d738-618965d7b3d44bf58c00664087392002 on tablet 618965d7b3d44bf58c00664087392002 with peer fa2c654d61eb4d32967da7ea2374d738
I20240501 19:56:06.514341 21314 raft_consensus.cc:1203] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:06.515609 21506 raft_consensus.cc:2932] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:06.516198 21314 raft_consensus.cc:2932] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:06.519016 19517 catalog_manager.cc:5484] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: config changed from index -1 to 4, VOTER fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:06.534260 19797 consensus_queue.cc:237] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
W20240501 19:56:06.538930 19732 consensus_peers.cc:485] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 -> Peer 2d6860caa69249c1837378d6f5247099 (127.17.65.4:42703): Couldn't send request to peer 2d6860caa69249c1837378d6f5247099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afb520db25524bc0b8deee1f443f6a44. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.104068 21531 ts_tablet_manager.cc:902] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: Initiating tablet copy from peer 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:07.104744 21531 tablet_copy_client.cc:308] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:42021
I20240501 19:56:07.108234 19817 tablet_copy_service.cc:140] P 1af1fefd33104e7bad976e2bd786c0b6: Received BeginTabletCopySession request for tablet afb520db25524bc0b8deee1f443f6a44 from peer 2d6860caa69249c1837378d6f5247099 ({username='slave'} at 127.17.65.4:55635)
I20240501 19:56:07.108323 19817 tablet_copy_service.cc:161] P 1af1fefd33104e7bad976e2bd786c0b6: Beginning new tablet copy session on tablet afb520db25524bc0b8deee1f443f6a44 from peer 2d6860caa69249c1837378d6f5247099 at {username='slave'} at 127.17.65.4:55635: session id = 2d6860caa69249c1837378d6f5247099-afb520db25524bc0b8deee1f443f6a44
I20240501 19:56:07.109030 19817 tablet_copy_source_session.cc:205] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.111825 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb520db25524bc0b8deee1f443f6a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.116586 21531 tablet_copy_client.cc:783] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.116778 21531 tablet_copy_client.cc:647] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.118109 21531 tablet_copy_client.cc:523] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.119486 21531 tablet_bootstrap.cc:492] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: Bootstrap starting.
I20240501 19:56:07.124228 21531 log.cc:824] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:07.141929 21531 tablet_bootstrap.cc:492] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:07.142248 21531 tablet_bootstrap.cc:492] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: Bootstrap complete.
I20240501 19:56:07.142494 21531 ts_tablet_manager.cc:1367] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.000s
I20240501 19:56:07.144253 21531 raft_consensus.cc:348] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:07.144766 21531 raft_consensus.cc:729] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6860caa69249c1837378d6f5247099, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.145040 21531 consensus_queue.cc:260] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:07.145742 21531 ts_tablet_manager.cc:1398] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 19:56:07.146354 19817 tablet_copy_service.cc:342] P 1af1fefd33104e7bad976e2bd786c0b6: Request end of tablet copy session 2d6860caa69249c1837378d6f5247099-afb520db25524bc0b8deee1f443f6a44 received from {username='slave'} at 127.17.65.4:55635
I20240501 19:56:07.146448 19817 tablet_copy_service.cc:433] P 1af1fefd33104e7bad976e2bd786c0b6: ending tablet copy session 2d6860caa69249c1837378d6f5247099-afb520db25524bc0b8deee1f443f6a44 on tablet afb520db25524bc0b8deee1f443f6a44 with peer 2d6860caa69249c1837378d6f5247099
I20240501 19:56:07.636771 20921 raft_consensus.cc:1203] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:07.637959 21541 raft_consensus.cc:2932] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:07.638455 20921 raft_consensus.cc:2932] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:07.639374 21541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52784 (request call id 0) took 1105 ms. Trace:
I20240501 19:56:07.639482 21541 rpcz_store.cc:276] 0501 19:56:06.533891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:56:06.533975 (+ 84us) service_pool.cc:225] Handling call
0501 19:56:06.534688 (+ 713us) log.cc:842] Serialized 254 byte log entry
0501 19:56:07.639358 (+1104670us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":38,"raft.queue_time_us":58,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":124,"wal-append.run_cpu_time_us":328,"wal-append.run_wall_time_us":1000360}
I20240501 19:56:07.641582 19520 catalog_manager.cc:5484] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 reported cstate change: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:07.657426 19666 consensus_queue.cc:237] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
W20240501 19:56:07.661818 19600 consensus_peers.cc:485] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 -> Peer 5bdf040be08b4aa292256860320503f7 (127.17.65.6:42633): Couldn't send request to peer 5bdf040be08b4aa292256860320503f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089c2d27c8c2479e8efafa68ebf12969. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:08.204293 21569 ts_tablet_manager.cc:902] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: Initiating tablet copy from peer 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:08.204984 21569 tablet_copy_client.cc:308] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:36451
I20240501 19:56:08.208498 19686 tablet_copy_service.cc:140] P 002bd5b7fe4c40169448f30e32ec54c6: Received BeginTabletCopySession request for tablet 089c2d27c8c2479e8efafa68ebf12969 from peer 5bdf040be08b4aa292256860320503f7 ({username='slave'} at 127.17.65.6:55095)
I20240501 19:56:08.208595 19686 tablet_copy_service.cc:161] P 002bd5b7fe4c40169448f30e32ec54c6: Beginning new tablet copy session on tablet 089c2d27c8c2479e8efafa68ebf12969 from peer 5bdf040be08b4aa292256860320503f7 at {username='slave'} at 127.17.65.6:55095: session id = 5bdf040be08b4aa292256860320503f7-089c2d27c8c2479e8efafa68ebf12969
I20240501 19:56:08.209455 19686 tablet_copy_source_session.cc:205] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:08.212555 21569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089c2d27c8c2479e8efafa68ebf12969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.217404 21569 tablet_copy_client.cc:783] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:08.217592 21569 tablet_copy_client.cc:647] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:08.219000 21569 tablet_copy_client.cc:523] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:08.220464 21569 tablet_bootstrap.cc:492] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: Bootstrap starting.
I20240501 19:56:08.225193 21569 log.cc:824] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:08.243335 21569 tablet_bootstrap.cc:492] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:08.243747 21569 tablet_bootstrap.cc:492] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: Bootstrap complete.
I20240501 19:56:08.244022 21569 ts_tablet_manager.cc:1367] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: Time spent bootstrapping tablet: real 0.024s user 0.024s sys 0.000s
I20240501 19:56:08.245942 21569 raft_consensus.cc:348] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:08.246541 21569 raft_consensus.cc:729] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdf040be08b4aa292256860320503f7, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.246840 21569 consensus_queue.cc:260] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:08.247658 21569 ts_tablet_manager.cc:1398] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240501 19:56:08.248422 19686 tablet_copy_service.cc:342] P 002bd5b7fe4c40169448f30e32ec54c6: Request end of tablet copy session 5bdf040be08b4aa292256860320503f7-089c2d27c8c2479e8efafa68ebf12969 received from {username='slave'} at 127.17.65.6:55095
I20240501 19:56:08.248517 19686 tablet_copy_service.cc:433] P 002bd5b7fe4c40169448f30e32ec54c6: ending tablet copy session 5bdf040be08b4aa292256860320503f7-089c2d27c8c2479e8efafa68ebf12969 on tablet 089c2d27c8c2479e8efafa68ebf12969 with peer 5bdf040be08b4aa292256860320503f7
I20240501 19:56:08.765233 21183 raft_consensus.cc:1203] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.766618 21579 raft_consensus.cc:2932] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bdf040be08b4aa292256860320503f7 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:08.767189 21183 raft_consensus.cc:2932] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bdf040be08b4aa292256860320503f7 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:08.774391 21579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60140 (request call id 0) took 1117 ms. Trace:
I20240501 19:56:08.774564 21579 rpcz_store.cc:276] 0501 19:56:07.657127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:56:07.657182 (+ 55us) service_pool.cc:225] Handling call
0501 19:56:07.657885 (+ 703us) log.cc:842] Serialized 254 byte log entry
0501 19:56:08.774369 (+1116484us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":44,"raft.queue_time_us":104,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":128,"thread_start_us":90,"threads_started":2,"wal-append.queue_time_us":116,"wal-append.run_cpu_time_us":303,"wal-append.run_wall_time_us":1000381}
I20240501 19:56:08.775988 19517 catalog_manager.cc:5484] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 reported cstate change: config changed from index -1 to 4, VOTER 5bdf040be08b4aa292256860320503f7 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:08.799713 19797 consensus_queue.cc:237] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
W20240501 19:56:08.804729 19730 consensus_peers.cc:485] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 -> Peer b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3:44125): Couldn't send request to peer b01a4a9860ce4bff90c3dc414228d44e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e05bb1fdf29d48b99d3aa029e315a519. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:09.215050 21606 ts_tablet_manager.cc:902] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: Initiating tablet copy from peer 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:09.215770 21606 tablet_copy_client.cc:308] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:42021
I20240501 19:56:09.219396 19817 tablet_copy_service.cc:140] P 1af1fefd33104e7bad976e2bd786c0b6: Received BeginTabletCopySession request for tablet e05bb1fdf29d48b99d3aa029e315a519 from peer b01a4a9860ce4bff90c3dc414228d44e ({username='slave'} at 127.17.65.3:39347)
I20240501 19:56:09.219480 19817 tablet_copy_service.cc:161] P 1af1fefd33104e7bad976e2bd786c0b6: Beginning new tablet copy session on tablet e05bb1fdf29d48b99d3aa029e315a519 from peer b01a4a9860ce4bff90c3dc414228d44e at {username='slave'} at 127.17.65.3:39347: session id = b01a4a9860ce4bff90c3dc414228d44e-e05bb1fdf29d48b99d3aa029e315a519
I20240501 19:56:09.220191 19817 tablet_copy_source_session.cc:205] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:09.223160 21606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05bb1fdf29d48b99d3aa029e315a519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.229013 21606 tablet_copy_client.cc:783] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:09.229183 21606 tablet_copy_client.cc:647] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:09.230553 21606 tablet_copy_client.cc:523] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:09.283728 21606 tablet_bootstrap.cc:492] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: Bootstrap starting.
I20240501 19:56:09.288982 21606 log.cc:824] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:09.310703 21606 tablet_bootstrap.cc:492] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:09.311154 21606 tablet_bootstrap.cc:492] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: Bootstrap complete.
I20240501 19:56:09.311448 21606 ts_tablet_manager.cc:1367] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.005s
I20240501 19:56:09.313261 21606 raft_consensus.cc:348] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:09.313766 21606 raft_consensus.cc:729] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01a4a9860ce4bff90c3dc414228d44e, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.314036 21606 consensus_queue.cc:260] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:09.314719 21606 ts_tablet_manager.cc:1398] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:09.315420 19817 tablet_copy_service.cc:342] P 1af1fefd33104e7bad976e2bd786c0b6: Request end of tablet copy session b01a4a9860ce4bff90c3dc414228d44e-e05bb1fdf29d48b99d3aa029e315a519 received from {username='slave'} at 127.17.65.3:39347
I20240501 19:56:09.315521 19817 tablet_copy_service.cc:433] P 1af1fefd33104e7bad976e2bd786c0b6: ending tablet copy session b01a4a9860ce4bff90c3dc414228d44e-e05bb1fdf29d48b99d3aa029e315a519 on tablet e05bb1fdf29d48b99d3aa029e315a519 with peer b01a4a9860ce4bff90c3dc414228d44e
I20240501 19:56:09.637986 20790 raft_consensus.cc:1203] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:09.639122 21615 raft_consensus.cc:2932] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:09.639680 20790 raft_consensus.cc:2932] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:09.687213 19520 catalog_manager.cc:5484] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: config changed from index -1 to 4, VOTER b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:09.705775 19666 consensus_queue.cc:237] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
W20240501 19:56:09.709976 19601 consensus_peers.cc:485] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 -> Peer fd3145d4045844a99067befa34a077be (127.17.65.5:41697): Couldn't send request to peer fd3145d4045844a99067befa34a077be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c57a563e77234a7b99dc890e2d5b2fa9. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:10.105389 21640 ts_tablet_manager.cc:902] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: Initiating tablet copy from peer 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:10.105844 21640 tablet_copy_client.cc:308] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:36451
I20240501 19:56:10.109364 19686 tablet_copy_service.cc:140] P 002bd5b7fe4c40169448f30e32ec54c6: Received BeginTabletCopySession request for tablet c57a563e77234a7b99dc890e2d5b2fa9 from peer fd3145d4045844a99067befa34a077be ({username='slave'} at 127.17.65.5:56079)
I20240501 19:56:10.109484 19686 tablet_copy_service.cc:161] P 002bd5b7fe4c40169448f30e32ec54c6: Beginning new tablet copy session on tablet c57a563e77234a7b99dc890e2d5b2fa9 from peer fd3145d4045844a99067befa34a077be at {username='slave'} at 127.17.65.5:56079: session id = fd3145d4045844a99067befa34a077be-c57a563e77234a7b99dc890e2d5b2fa9
I20240501 19:56:10.110209 19686 tablet_copy_source_session.cc:205] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:10.110991 21640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57a563e77234a7b99dc890e2d5b2fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:10.113046 21640 tablet_copy_client.cc:783] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:10.113197 21640 tablet_copy_client.cc:647] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:10.114179 21640 tablet_copy_client.cc:523] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:10.182348 21640 tablet_bootstrap.cc:492] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: Bootstrap starting.
I20240501 19:56:10.199931 21640 tablet_bootstrap.cc:492] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:10.200227 21640 tablet_bootstrap.cc:492] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: Bootstrap complete.
I20240501 19:56:10.200395 21640 ts_tablet_manager.cc:1367] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20240501 19:56:10.200942 21640 raft_consensus.cc:348] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:10.201170 21640 raft_consensus.cc:729] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3145d4045844a99067befa34a077be, State: Initialized, Role: FOLLOWER
I20240501 19:56:10.201290 21640 consensus_queue.cc:260] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:10.201624 21640 ts_tablet_manager.cc:1398] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 19:56:10.202268 19686 tablet_copy_service.cc:342] P 002bd5b7fe4c40169448f30e32ec54c6: Request end of tablet copy session fd3145d4045844a99067befa34a077be-c57a563e77234a7b99dc890e2d5b2fa9 received from {username='slave'} at 127.17.65.5:56079
I20240501 19:56:10.202405 19686 tablet_copy_service.cc:433] P 002bd5b7fe4c40169448f30e32ec54c6: ending tablet copy session fd3145d4045844a99067befa34a077be-c57a563e77234a7b99dc890e2d5b2fa9 on tablet c57a563e77234a7b99dc890e2d5b2fa9 with peer fd3145d4045844a99067befa34a077be
I20240501 19:56:10.687460 21052 raft_consensus.cc:1203] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:10.688282 21648 raft_consensus.cc:2932] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd3145d4045844a99067befa34a077be (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:10.688735 21052 raft_consensus.cc:2932] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd3145d4045844a99067befa34a077be (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:10.691962 19517 catalog_manager.cc:5484] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: config changed from index -1 to 4, VOTER fd3145d4045844a99067befa34a077be (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:10.708284 19666 consensus_queue.cc:237] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
W20240501 19:56:10.709537 19601 consensus_peers.cc:485] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 -> Peer fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7:34015): Couldn't send request to peer fa2c654d61eb4d32967da7ea2374d738. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68a0476d7a0c4d1f8dda6182816f4a17. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:11.220311 21672 ts_tablet_manager.cc:902] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: Initiating tablet copy from peer 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:11.220737 21672 tablet_copy_client.cc:308] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:36451
I20240501 19:56:11.221213 19686 tablet_copy_service.cc:140] P 002bd5b7fe4c40169448f30e32ec54c6: Received BeginTabletCopySession request for tablet 68a0476d7a0c4d1f8dda6182816f4a17 from peer fa2c654d61eb4d32967da7ea2374d738 ({username='slave'} at 127.17.65.7:34891)
I20240501 19:56:11.221314 19686 tablet_copy_service.cc:161] P 002bd5b7fe4c40169448f30e32ec54c6: Beginning new tablet copy session on tablet 68a0476d7a0c4d1f8dda6182816f4a17 from peer fa2c654d61eb4d32967da7ea2374d738 at {username='slave'} at 127.17.65.7:34891: session id = fa2c654d61eb4d32967da7ea2374d738-68a0476d7a0c4d1f8dda6182816f4a17
I20240501 19:56:11.222015 19686 tablet_copy_source_session.cc:205] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:11.222833 21672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a0476d7a0c4d1f8dda6182816f4a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:11.224498 21672 tablet_copy_client.cc:783] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:11.224639 21672 tablet_copy_client.cc:647] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:11.225826 21672 tablet_copy_client.cc:523] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:11.226831 21672 tablet_bootstrap.cc:492] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: Bootstrap starting.
I20240501 19:56:11.243763 21672 tablet_bootstrap.cc:492] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:11.244092 21672 tablet_bootstrap.cc:492] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: Bootstrap complete.
I20240501 19:56:11.244261 21672 ts_tablet_manager.cc:1367] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240501 19:56:11.244825 21672 raft_consensus.cc:348] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:11.245055 21672 raft_consensus.cc:729] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2c654d61eb4d32967da7ea2374d738, State: Initialized, Role: FOLLOWER
I20240501 19:56:11.245153 21672 consensus_queue.cc:260] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:11.245568 21672 ts_tablet_manager.cc:1398] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:11.246173 19686 tablet_copy_service.cc:342] P 002bd5b7fe4c40169448f30e32ec54c6: Request end of tablet copy session fa2c654d61eb4d32967da7ea2374d738-68a0476d7a0c4d1f8dda6182816f4a17 received from {username='slave'} at 127.17.65.7:34891
I20240501 19:56:11.246258 19686 tablet_copy_service.cc:433] P 002bd5b7fe4c40169448f30e32ec54c6: ending tablet copy session fa2c654d61eb4d32967da7ea2374d738-68a0476d7a0c4d1f8dda6182816f4a17 on tablet 68a0476d7a0c4d1f8dda6182816f4a17 with peer fa2c654d61eb4d32967da7ea2374d738
I20240501 19:56:11.724195 21314 raft_consensus.cc:1203] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:11.725121 21679 raft_consensus.cc:2932] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:11.725523 21314 raft_consensus.cc:2932] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:11.726595 21679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60168 (request call id 0) took 1018 ms. Trace:
I20240501 19:56:11.726707 21679 rpcz_store.cc:276] 0501 19:56:10.707963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:56:10.708001 (+ 38us) service_pool.cc:225] Handling call
0501 19:56:10.708679 (+ 678us) log.cc:842] Serialized 254 byte log entry
0501 19:56:11.726579 (+1017900us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":44,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":119,"wal-append.run_cpu_time_us":350,"wal-append.run_wall_time_us":1000416}
I20240501 19:56:11.728987 19524 catalog_manager.cc:5484] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 reported cstate change: config changed from index -1 to 4, VOTER fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:11.744187 19666 consensus_queue.cc:237] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
W20240501 19:56:11.748107 19601 consensus_peers.cc:485] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 -> Peer 2d6860caa69249c1837378d6f5247099 (127.17.65.4:42703): Couldn't send request to peer 2d6860caa69249c1837378d6f5247099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2de3fc3f434240748a8f75260808f9e5. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:12.162415 21706 ts_tablet_manager.cc:902] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Initiating tablet copy from peer 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:12.162904 21706 tablet_copy_client.cc:308] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:36451
I20240501 19:56:12.166445 19686 tablet_copy_service.cc:140] P 002bd5b7fe4c40169448f30e32ec54c6: Received BeginTabletCopySession request for tablet 2de3fc3f434240748a8f75260808f9e5 from peer 2d6860caa69249c1837378d6f5247099 ({username='slave'} at 127.17.65.4:38495)
I20240501 19:56:12.166563 19686 tablet_copy_service.cc:161] P 002bd5b7fe4c40169448f30e32ec54c6: Beginning new tablet copy session on tablet 2de3fc3f434240748a8f75260808f9e5 from peer 2d6860caa69249c1837378d6f5247099 at {username='slave'} at 127.17.65.4:38495: session id = 2d6860caa69249c1837378d6f5247099-2de3fc3f434240748a8f75260808f9e5
I20240501 19:56:12.167272 19686 tablet_copy_source_session.cc:205] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:12.168046 21706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de3fc3f434240748a8f75260808f9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:12.169827 21706 tablet_copy_client.cc:783] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:12.169986 21706 tablet_copy_client.cc:647] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:12.170967 21706 tablet_copy_client.cc:523] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:12.172003 21706 tablet_bootstrap.cc:492] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Bootstrap starting.
I20240501 19:56:12.188546 21706 tablet_bootstrap.cc:492] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:12.188848 21706 tablet_bootstrap.cc:492] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Bootstrap complete.
I20240501 19:56:12.189018 21706 ts_tablet_manager.cc:1367] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20240501 19:56:12.189520 21706 raft_consensus.cc:348] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:12.189738 21706 raft_consensus.cc:729] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6860caa69249c1837378d6f5247099, State: Initialized, Role: FOLLOWER
I20240501 19:56:12.189832 21706 consensus_queue.cc:260] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:12.190171 21706 ts_tablet_manager.cc:1398] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 19:56:12.190766 19686 tablet_copy_service.cc:342] P 002bd5b7fe4c40169448f30e32ec54c6: Request end of tablet copy session 2d6860caa69249c1837378d6f5247099-2de3fc3f434240748a8f75260808f9e5 received from {username='slave'} at 127.17.65.4:38495
I20240501 19:56:12.190846 19686 tablet_copy_service.cc:433] P 002bd5b7fe4c40169448f30e32ec54c6: ending tablet copy session 2d6860caa69249c1837378d6f5247099-2de3fc3f434240748a8f75260808f9e5 on tablet 2de3fc3f434240748a8f75260808f9e5 with peer 2d6860caa69249c1837378d6f5247099
I20240501 19:56:12.736913 20921 raft_consensus.cc:1203] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:12.737846 21715 raft_consensus.cc:2932] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:12.738284 20921 raft_consensus.cc:2932] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:12.741451 19524 catalog_manager.cc:5484] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:12.757441 19797 consensus_queue.cc:237] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
W20240501 19:56:12.761773 19731 consensus_peers.cc:485] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 -> Peer 5bdf040be08b4aa292256860320503f7 (127.17.65.6:42633): Couldn't send request to peer 5bdf040be08b4aa292256860320503f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32c3ffbbb4d140bbae02fdb2fed1e109. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:13.335733 21741 ts_tablet_manager.cc:902] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: Initiating tablet copy from peer 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:13.336279 21741 tablet_copy_client.cc:308] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:42021
I20240501 19:56:13.340204 19817 tablet_copy_service.cc:140] P 1af1fefd33104e7bad976e2bd786c0b6: Received BeginTabletCopySession request for tablet 32c3ffbbb4d140bbae02fdb2fed1e109 from peer 5bdf040be08b4aa292256860320503f7 ({username='slave'} at 127.17.65.6:36197)
I20240501 19:56:13.340374 19817 tablet_copy_service.cc:161] P 1af1fefd33104e7bad976e2bd786c0b6: Beginning new tablet copy session on tablet 32c3ffbbb4d140bbae02fdb2fed1e109 from peer 5bdf040be08b4aa292256860320503f7 at {username='slave'} at 127.17.65.6:36197: session id = 5bdf040be08b4aa292256860320503f7-32c3ffbbb4d140bbae02fdb2fed1e109
I20240501 19:56:13.341747 19817 tablet_copy_source_session.cc:205] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:13.343020 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c3ffbbb4d140bbae02fdb2fed1e109. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:13.345122 21741 tablet_copy_client.cc:783] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:13.345263 21741 tablet_copy_client.cc:647] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:13.346313 21741 tablet_copy_client.cc:523] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:13.347381 21741 tablet_bootstrap.cc:492] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: Bootstrap starting.
I20240501 19:56:13.363266 21741 tablet_bootstrap.cc:492] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:13.363705 21741 tablet_bootstrap.cc:492] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: Bootstrap complete.
I20240501 19:56:13.363895 21741 ts_tablet_manager.cc:1367] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20240501 19:56:13.364488 21741 raft_consensus.cc:348] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:13.364768 21741 raft_consensus.cc:729] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdf040be08b4aa292256860320503f7, State: Initialized, Role: FOLLOWER
I20240501 19:56:13.364886 21741 consensus_queue.cc:260] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:13.365247 21741 ts_tablet_manager.cc:1398] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:13.365895 19817 tablet_copy_service.cc:342] P 1af1fefd33104e7bad976e2bd786c0b6: Request end of tablet copy session 5bdf040be08b4aa292256860320503f7-32c3ffbbb4d140bbae02fdb2fed1e109 received from {username='slave'} at 127.17.65.6:36197
I20240501 19:56:13.365995 19817 tablet_copy_service.cc:433] P 1af1fefd33104e7bad976e2bd786c0b6: ending tablet copy session 5bdf040be08b4aa292256860320503f7-32c3ffbbb4d140bbae02fdb2fed1e109 on tablet 32c3ffbbb4d140bbae02fdb2fed1e109 with peer 5bdf040be08b4aa292256860320503f7
I20240501 19:56:13.804883 21183 raft_consensus.cc:1203] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:13.805620 21505 raft_consensus.cc:2932] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bdf040be08b4aa292256860320503f7 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:13.806046 21183 raft_consensus.cc:2932] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bdf040be08b4aa292256860320503f7 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:13.807180 21505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52812 (request call id 0) took 1050 ms. Trace:
I20240501 19:56:13.807246 21505 rpcz_store.cc:276] 0501 19:56:12.757049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:56:12.757122 (+ 73us) service_pool.cc:225] Handling call
0501 19:56:12.757915 (+ 793us) log.cc:842] Serialized 254 byte log entry
0501 19:56:13.807164 (+1049249us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":45,"raft.queue_time_us":37,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"spinlock_wait_cycles":1408,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":139,"wal-append.run_cpu_time_us":319,"wal-append.run_wall_time_us":1000424}
I20240501 19:56:13.808769 19520 catalog_manager.cc:5484] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 reported cstate change: config changed from index -1 to 4, VOTER 5bdf040be08b4aa292256860320503f7 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:13.830981 19797 consensus_queue.cc:237] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
W20240501 19:56:13.832312 19730 consensus_peers.cc:485] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 -> Peer b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3:44125): Couldn't send request to peer b01a4a9860ce4bff90c3dc414228d44e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecdb17582af74a12b63acbbe2560166d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:14.288573 21774 ts_tablet_manager.cc:902] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: Initiating tablet copy from peer 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:14.289049 21774 tablet_copy_client.cc:308] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:42021
I20240501 19:56:14.289563 19817 tablet_copy_service.cc:140] P 1af1fefd33104e7bad976e2bd786c0b6: Received BeginTabletCopySession request for tablet ecdb17582af74a12b63acbbe2560166d from peer b01a4a9860ce4bff90c3dc414228d44e ({username='slave'} at 127.17.65.3:39347)
I20240501 19:56:14.289665 19817 tablet_copy_service.cc:161] P 1af1fefd33104e7bad976e2bd786c0b6: Beginning new tablet copy session on tablet ecdb17582af74a12b63acbbe2560166d from peer b01a4a9860ce4bff90c3dc414228d44e at {username='slave'} at 127.17.65.3:39347: session id = b01a4a9860ce4bff90c3dc414228d44e-ecdb17582af74a12b63acbbe2560166d
I20240501 19:56:14.290370 19817 tablet_copy_source_session.cc:205] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:14.291126 21774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecdb17582af74a12b63acbbe2560166d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:14.292762 21774 tablet_copy_client.cc:783] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:14.292903 21774 tablet_copy_client.cc:647] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:14.293888 21774 tablet_copy_client.cc:523] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:14.294759 21774 tablet_bootstrap.cc:492] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: Bootstrap starting.
I20240501 19:56:14.297436 21774 tablet_bootstrap.cc:492] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:14.297652 21774 tablet_bootstrap.cc:492] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: Bootstrap complete.
I20240501 19:56:14.297799 21774 ts_tablet_manager.cc:1367] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:14.298280 21774 raft_consensus.cc:348] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } }
I20240501 19:56:14.298491 21774 raft_consensus.cc:729] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01a4a9860ce4bff90c3dc414228d44e, State: Initialized, Role: FOLLOWER
I20240501 19:56:14.298586 21774 consensus_queue.cc:260] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:14.298880 21774 ts_tablet_manager.cc:1398] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:14.299312 19817 tablet_copy_service.cc:342] P 1af1fefd33104e7bad976e2bd786c0b6: Request end of tablet copy session b01a4a9860ce4bff90c3dc414228d44e-ecdb17582af74a12b63acbbe2560166d received from {username='slave'} at 127.17.65.3:39347
I20240501 19:56:14.299376 19817 tablet_copy_service.cc:433] P 1af1fefd33104e7bad976e2bd786c0b6: ending tablet copy session b01a4a9860ce4bff90c3dc414228d44e-ecdb17582af74a12b63acbbe2560166d on tablet ecdb17582af74a12b63acbbe2560166d with peer b01a4a9860ce4bff90c3dc414228d44e
I20240501 19:56:14.898149 20790 raft_consensus.cc:1203] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:14.899087 21782 raft_consensus.cc:2932] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:14.899473 20790 raft_consensus.cc:2932] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:14.900527 21782 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52824 (request call id 0) took 1069 ms. Trace:
I20240501 19:56:14.900599 21782 rpcz_store.cc:276] 0501 19:56:13.830646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:56:13.830700 (+ 54us) service_pool.cc:225] Handling call
0501 19:56:13.831384 (+ 684us) log.cc:842] Serialized 254 byte log entry
0501 19:56:14.900510 (+1069126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":37,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":141,"spinlock_wait_cycles":2432,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":160,"wal-append.run_cpu_time_us":335,"wal-append.run_wall_time_us":1000398}
I20240501 19:56:14.901918 19524 catalog_manager.cc:5484] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 reported cstate change: config changed from index -1 to 2, VOTER b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "1af1fefd33104e7bad976e2bd786c0b6" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:14.954386 19777 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:14.955183 21163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:14.955832 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:14.960701 20770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:14.967489 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:14.967597 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:14.970060 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.003175 19797 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ecdb17582af74a12b63acbbe2560166d"
dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39466
I20240501 19:56:15.003365 19797 raft_consensus.cc:595] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:15.050047 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.051813 19777 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:15.051905 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:15.056753 21163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.057343 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.065191 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.071316 20770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.152622 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:15.153926 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.154101 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.164750 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.165097 20770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.165165 21163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.183750 19777 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:15.203676 19797 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32c3ffbbb4d140bbae02fdb2fed1e109"
dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39508
I20240501 19:56:15.203797 19797 raft_consensus.cc:595] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:15.253262 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.254191 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:15.255632 19777 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:15.256613 20770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.257323 21163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.263491 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.291597 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.346063 21505 raft_consensus.cc:979] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: : Instructing follower 5bdf040be08b4aa292256860320503f7 to start an election
I20240501 19:56:15.346179 21505 raft_consensus.cc:1067] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Signalling peer 5bdf040be08b4aa292256860320503f7 to start an election
I20240501 19:56:15.346835 21183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32c3ffbbb4d140bbae02fdb2fed1e109"
dest_uuid: "5bdf040be08b4aa292256860320503f7"
from {username='slave'} at 127.17.65.2:56969
I20240501 19:56:15.347021 21183 raft_consensus.cc:482] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:15.347172 21183 raft_consensus.cc:3037] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:15.348457 21183 raft_consensus.cc:504] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:15.349404 21183 leader_election.cc:288] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [CANDIDATE]: Term 2 election: Requested vote from peers 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:15.349891 19797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c3ffbbb4d140bbae02fdb2fed1e109" candidate_uuid: "5bdf040be08b4aa292256860320503f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
I20240501 19:56:15.350068 19797 raft_consensus.cc:3032] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:15.350122 19797 raft_consensus.cc:729] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:15.350282 19797 consensus_queue.cc:260] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:15.350564 19797 raft_consensus.cc:3037] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:15.378487 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.378991 21163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.380061 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:15.386118 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.386574 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.391671 20770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.413522 19797 raft_consensus.cc:2454] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bdf040be08b4aa292256860320503f7 in term 2.
I20240501 19:56:15.413990 21116 leader_election.cc:302] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6, 5bdf040be08b4aa292256860320503f7; no voters:
I20240501 19:56:15.415582 19777 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:15.419903 21982 raft_consensus.cc:2781] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:15.420080 21982 raft_consensus.cc:686] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 2 LEADER]: Becoming Leader. State: Replica: 5bdf040be08b4aa292256860320503f7, State: Running, Role: LEADER
I20240501 19:56:15.420416 21982 consensus_queue.cc:237] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:15.421767 21782 raft_consensus.cc:979] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: : Instructing follower b01a4a9860ce4bff90c3dc414228d44e to start an election
I20240501 19:56:15.421866 21782 raft_consensus.cc:1067] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Signalling peer b01a4a9860ce4bff90c3dc414228d44e to start an election
I20240501 19:56:15.422354 20790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ecdb17582af74a12b63acbbe2560166d"
dest_uuid: "b01a4a9860ce4bff90c3dc414228d44e"
from {username='slave'} at 127.17.65.2:46075
I20240501 19:56:15.422538 20790 raft_consensus.cc:482] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:15.422632 20790 raft_consensus.cc:3037] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:15.423739 20790 raft_consensus.cc:504] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:15.424614 20790 leader_election.cc:288] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [CANDIDATE]: Term 2 election: Requested vote from peers 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:15.424907 19797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecdb17582af74a12b63acbbe2560166d" candidate_uuid: "b01a4a9860ce4bff90c3dc414228d44e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
I20240501 19:56:15.425012 19797 raft_consensus.cc:3032] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:15.425060 19797 raft_consensus.cc:729] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:15.425212 19797 consensus_queue.cc:260] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:15.425297 19797 raft_consensus.cc:3037] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:15.425212 19520 catalog_manager.cc:5484] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 reported cstate change: term changed from 1 to 2, leader changed from 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) to 5bdf040be08b4aa292256860320503f7 (127.17.65.6). New cstate: current_term: 2 leader_uuid: "5bdf040be08b4aa292256860320503f7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:15.426182 19797 raft_consensus.cc:2454] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01a4a9860ce4bff90c3dc414228d44e in term 2.
I20240501 19:56:15.426528 20723 leader_election.cc:302] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6, b01a4a9860ce4bff90c3dc414228d44e; no voters:
I20240501 19:56:15.426851 21986 raft_consensus.cc:2781] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:15.427076 21986 raft_consensus.cc:686] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 2 LEADER]: Becoming Leader. State: Replica: b01a4a9860ce4bff90c3dc414228d44e, State: Running, Role: LEADER
I20240501 19:56:15.427381 21986 consensus_queue.cc:237] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:15.429395 19520 catalog_manager.cc:5484] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e reported cstate change: term changed from 1 to 2, leader changed from 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) to b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3). New cstate: current_term: 2 leader_uuid: "b01a4a9860ce4bff90c3dc414228d44e" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:15.438972 19666 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "618965d7b3d44bf58c00664087392002"
dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39334
I20240501 19:56:15.439163 19666 raft_consensus.cc:595] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:15.491928 19777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:15.492604 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.495020 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.496673 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.504068 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:15.504761 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.505263 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.598807 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:15.600646 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.603853 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.607571 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.607942 19777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:15.613919 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.614586 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.644559 19666 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c57a563e77234a7b99dc890e2d5b2fa9"
dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39368
I20240501 19:56:15.644698 19666 raft_consensus.cc:595] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:15.692112 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.693147 19645 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:15.695188 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.696363 21294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.700915 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.701400 19777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:15.723534 21032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.772797 21715 raft_consensus.cc:979] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: : Instructing follower fd3145d4045844a99067befa34a077be to start an election
I20240501 19:56:15.772922 21715 raft_consensus.cc:1067] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Signalling peer fd3145d4045844a99067befa34a077be to start an election
I20240501 19:56:15.773500 21052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c57a563e77234a7b99dc890e2d5b2fa9"
dest_uuid: "fd3145d4045844a99067befa34a077be"
from {username='slave'} at 127.17.65.1:37287
I20240501 19:56:15.773691 21052 raft_consensus.cc:482] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:15.773805 21052 raft_consensus.cc:3037] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:15.774854 21052 raft_consensus.cc:504] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:15.775693 21052 leader_election.cc:288] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [CANDIDATE]: Term 2 election: Requested vote from peers 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:15.776139 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57a563e77234a7b99dc890e2d5b2fa9" candidate_uuid: "fd3145d4045844a99067befa34a077be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
I20240501 19:56:15.776315 19666 raft_consensus.cc:3032] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:15.776384 19666 raft_consensus.cc:729] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:15.776572 19666 consensus_queue.cc:260] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:15.776926 19666 raft_consensus.cc:3037] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:15.778084 19666 raft_consensus.cc:2454] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3145d4045844a99067befa34a077be in term 2.
I20240501 19:56:15.778503 20985 leader_election.cc:302] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6, fd3145d4045844a99067befa34a077be; no voters:
I20240501 19:56:15.778859 22123 raft_consensus.cc:2781] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:15.779055 22123 raft_consensus.cc:686] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 2 LEADER]: Becoming Leader. State: Replica: fd3145d4045844a99067befa34a077be, State: Running, Role: LEADER
I20240501 19:56:15.779366 22123 consensus_queue.cc:237] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:15.781831 19518 catalog_manager.cc:5484] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be reported cstate change: term changed from 1 to 2, leader changed from 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) to fd3145d4045844a99067befa34a077be (127.17.65.5). New cstate: current_term: 2 leader_uuid: "fd3145d4045844a99067befa34a077be" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:15.804591 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.805522 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.806283 19645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:15.807001 19777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:15.808557 22117 raft_consensus.cc:979] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: : Instructing follower fa2c654d61eb4d32967da7ea2374d738 to start an election
I20240501 19:56:15.808651 22117 raft_consensus.cc:1067] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Signalling peer fa2c654d61eb4d32967da7ea2374d738 to start an election
I20240501 19:56:15.809054 21314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "618965d7b3d44bf58c00664087392002"
dest_uuid: "fa2c654d61eb4d32967da7ea2374d738"
from {username='slave'} at 127.17.65.1:48091
I20240501 19:56:15.809224 21314 raft_consensus.cc:482] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:15.809312 21314 raft_consensus.cc:3037] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:15.810297 21314 raft_consensus.cc:504] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:15.811255 21314 leader_election.cc:288] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [CANDIDATE]: Term 2 election: Requested vote from peers 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:15.811751 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618965d7b3d44bf58c00664087392002" candidate_uuid: "fa2c654d61eb4d32967da7ea2374d738" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
I20240501 19:56:15.811870 19665 raft_consensus.cc:3032] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:15.811915 19665 raft_consensus.cc:729] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:15.812060 19665 consensus_queue.cc:260] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:15.812214 19665 raft_consensus.cc:3037] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:15.813129 19665 raft_consensus.cc:2454] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa2c654d61eb4d32967da7ea2374d738 in term 2.
I20240501 19:56:15.813479 21247 leader_election.cc:302] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6, fa2c654d61eb4d32967da7ea2374d738; no voters:
I20240501 19:56:15.813930 22143 raft_consensus.cc:2781] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:15.814085 22143 raft_consensus.cc:686] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 2 LEADER]: Becoming Leader. State: Replica: fa2c654d61eb4d32967da7ea2374d738, State: Running, Role: LEADER
I20240501 19:56:15.814630 22143 consensus_queue.cc:237] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:15.817445 19518 catalog_manager.cc:5484] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 reported cstate change: term changed from 1 to 2, leader changed from 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) to fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7). New cstate: current_term: 2 leader_uuid: "fa2c654d61eb4d32967da7ea2374d738" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:15.824031 21294 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.829321 21032 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.832181 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.857422 19665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68a0476d7a0c4d1f8dda6182816f4a17"
dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39404
I20240501 19:56:15.857582 19665 raft_consensus.cc:595] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:15.893069 19797 raft_consensus.cc:1261] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Refusing update from remote peer 5bdf040be08b4aa292256860320503f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:15.893738 21982 consensus_queue.cc:1035] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:15.914688 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:15.915907 19645 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:15.918329 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.919512 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.923341 19777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:15.926259 21294 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.943468 21032 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:15.977320 19797 raft_consensus.cc:1261] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Refusing update from remote peer b01a4a9860ce4bff90c3dc414228d44e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:15.977807 22197 consensus_queue.cc:1035] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:16.015898 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.016326 19645 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:16.018431 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.025532 21294 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.025831 21032 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.026614 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:16.037360 19777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:16.076578 19797 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ec069697dd9474fab37c77aa8abdb28"
dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39582
I20240501 19:56:16.076730 19797 raft_consensus.cc:595] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:16.114248 21715 raft_consensus.cc:979] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: : Instructing follower fa2c654d61eb4d32967da7ea2374d738 to start an election
I20240501 19:56:16.114349 21715 raft_consensus.cc:1067] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Signalling peer fa2c654d61eb4d32967da7ea2374d738 to start an election
I20240501 19:56:16.115382 21314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68a0476d7a0c4d1f8dda6182816f4a17"
dest_uuid: "fa2c654d61eb4d32967da7ea2374d738"
from {username='slave'} at 127.17.65.1:48091
I20240501 19:56:16.115537 21314 raft_consensus.cc:482] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:16.115597 21314 raft_consensus.cc:3037] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:16.116683 21314 raft_consensus.cc:504] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:16.116940 21314 leader_election.cc:288] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [CANDIDATE]: Term 2 election: Requested vote from peers 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:16.117307 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a0476d7a0c4d1f8dda6182816f4a17" candidate_uuid: "fa2c654d61eb4d32967da7ea2374d738" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
I20240501 19:56:16.117432 19665 raft_consensus.cc:3032] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:16.117484 19665 raft_consensus.cc:729] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:16.117623 19665 consensus_queue.cc:260] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:16.117775 19665 raft_consensus.cc:3037] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:16.118592 19665 raft_consensus.cc:2454] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa2c654d61eb4d32967da7ea2374d738 in term 2.
I20240501 19:56:16.118871 21247 leader_election.cc:302] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6, fa2c654d61eb4d32967da7ea2374d738; no voters:
I20240501 19:56:16.119069 22144 raft_consensus.cc:2781] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:16.119151 22144 raft_consensus.cc:686] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 2 LEADER]: Becoming Leader. State: Replica: fa2c654d61eb4d32967da7ea2374d738, State: Running, Role: LEADER
I20240501 19:56:16.119275 22144 consensus_queue.cc:237] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:16.121273 19517 catalog_manager.cc:5484] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 reported cstate change: term changed from 1 to 2, leader changed from 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) to fa2c654d61eb4d32967da7ea2374d738 (127.17.65.7). New cstate: current_term: 2 leader_uuid: "fa2c654d61eb4d32967da7ea2374d738" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:16.129006 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.130249 19777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:16.131060 19645 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:16.132092 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.137224 21032 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.148758 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:16.163173 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.193735 21782 raft_consensus.cc:979] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: : Instructing follower fd3145d4045844a99067befa34a077be to start an election
I20240501 19:56:16.193835 21782 raft_consensus.cc:1067] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Signalling peer fd3145d4045844a99067befa34a077be to start an election
I20240501 19:56:16.194204 21052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ec069697dd9474fab37c77aa8abdb28"
dest_uuid: "fd3145d4045844a99067befa34a077be"
from {username='slave'} at 127.17.65.2:43539
I20240501 19:56:16.194347 21052 raft_consensus.cc:482] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:16.194402 21052 raft_consensus.cc:3037] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:16.195444 21052 raft_consensus.cc:504] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:16.195781 21052 leader_election.cc:288] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [CANDIDATE]: Term 2 election: Requested vote from peers 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:16.196138 19797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec069697dd9474fab37c77aa8abdb28" candidate_uuid: "fd3145d4045844a99067befa34a077be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
I20240501 19:56:16.196252 19797 raft_consensus.cc:3032] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:16.196297 19797 raft_consensus.cc:729] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:16.196457 19797 consensus_queue.cc:260] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:16.196584 19797 raft_consensus.cc:3037] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:16.197453 19797 raft_consensus.cc:2454] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3145d4045844a99067befa34a077be in term 2.
I20240501 19:56:16.197760 20985 leader_election.cc:302] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6, fd3145d4045844a99067befa34a077be; no voters:
I20240501 19:56:16.197916 22123 raft_consensus.cc:2781] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:16.197993 22123 raft_consensus.cc:686] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 2 LEADER]: Becoming Leader. State: Replica: fd3145d4045844a99067befa34a077be, State: Running, Role: LEADER
I20240501 19:56:16.198133 22123 consensus_queue.cc:237] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:16.199635 19517 catalog_manager.cc:5484] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be reported cstate change: term changed from 1 to 2, leader changed from 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) to fd3145d4045844a99067befa34a077be (127.17.65.5). New cstate: current_term: 2 leader_uuid: "fd3145d4045844a99067befa34a077be" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:16.204185 19665 raft_consensus.cc:1261] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Refusing update from remote peer fd3145d4045844a99067befa34a077be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:16.204713 22123 consensus_queue.cc:1035] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [LEADER]: Connected to new peer: Peer: permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:16.235596 19777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:16.239995 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:16.241163 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.248672 19645 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:16.256940 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.256945 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.267835 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.304749 19665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2de3fc3f434240748a8f75260808f9e5"
dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39450
I20240501 19:56:16.304936 19665 raft_consensus.cc:595] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:16.359514 19645 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:16.359632 19777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:16.360244 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:16.364382 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.367795 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.374826 19665 raft_consensus.cc:1261] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Refusing update from remote peer fa2c654d61eb4d32967da7ea2374d738: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:16.375576 22144 consensus_queue.cc:1035] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:16.377750 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.396178 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.478240 19645 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:16.478991 20901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:16.479213 19777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:16.480140 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.482115 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.499291 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.500092 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.533530 19797 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e05bb1fdf29d48b99d3aa029e315a519"
dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39634
I20240501 19:56:16.533687 19797 raft_consensus.cc:595] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:16.537460 21715 raft_consensus.cc:979] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: : Instructing follower 2d6860caa69249c1837378d6f5247099 to start an election
I20240501 19:56:16.537585 21715 raft_consensus.cc:1067] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Signalling peer 2d6860caa69249c1837378d6f5247099 to start an election
I20240501 19:56:16.538096 20921 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2de3fc3f434240748a8f75260808f9e5"
dest_uuid: "2d6860caa69249c1837378d6f5247099"
from {username='slave'} at 127.17.65.1:54961
I20240501 19:56:16.538277 20921 raft_consensus.cc:482] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:16.538405 20921 raft_consensus.cc:3037] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:16.539465 20921 raft_consensus.cc:504] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:16.540407 20921 leader_election.cc:288] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [CANDIDATE]: Term 2 election: Requested vote from peers 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:16.540805 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de3fc3f434240748a8f75260808f9e5" candidate_uuid: "2d6860caa69249c1837378d6f5247099" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
I20240501 19:56:16.540920 19665 raft_consensus.cc:3032] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:16.540966 19665 raft_consensus.cc:729] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:16.541110 19665 consensus_queue.cc:260] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:16.541235 19665 raft_consensus.cc:3037] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:16.542001 19665 raft_consensus.cc:2454] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d6860caa69249c1837378d6f5247099 in term 2.
I20240501 19:56:16.542338 20854 leader_election.cc:302] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6, 2d6860caa69249c1837378d6f5247099; no voters:
I20240501 19:56:16.542683 22396 raft_consensus.cc:2781] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:16.542840 22396 raft_consensus.cc:686] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Becoming Leader. State: Replica: 2d6860caa69249c1837378d6f5247099, State: Running, Role: LEADER
I20240501 19:56:16.543195 22396 consensus_queue.cc:237] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:16.545356 19518 catalog_manager.cc:5484] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 reported cstate change: term changed from 1 to 2, leader changed from 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) to 2d6860caa69249c1837378d6f5247099 (127.17.65.4). New cstate: current_term: 2 leader_uuid: "2d6860caa69249c1837378d6f5247099" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:16.589243 19645 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:16.589881 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.591008 19777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:16.594121 20901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.594697 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.595703 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.601218 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.693557 19645 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:16.693852 20901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.695185 19777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:16.700994 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.702193 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.703569 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.715206 19665 raft_consensus.cc:1261] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Refusing update from remote peer fa2c654d61eb4d32967da7ea2374d738: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:16.715801 22144 consensus_queue.cc:1035] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:16.719540 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.747634 19797 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "afb520db25524bc0b8deee1f443f6a44"
dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39678
I20240501 19:56:16.747783 19797 raft_consensus.cc:595] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:16.796531 19645 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:16.798537 20901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.802191 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.802784 19777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:16.804507 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.813002 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.820602 19797 raft_consensus.cc:1261] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Refusing update from remote peer fd3145d4045844a99067befa34a077be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:16.821151 22124 consensus_queue.cc:1035] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:16.822022 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.905089 19777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:16.905992 19645 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:16.906967 20901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.910385 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.919178 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:16.929913 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.931651 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:16.959573 19665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "089c2d27c8c2479e8efafa68ebf12969"
dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39510
I20240501 19:56:16.959767 19665 raft_consensus.cc:595] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:17.008481 19645 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:17.009995 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:17.013574 20770 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:17.020474 20901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:17.026883 19777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:17.029148 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.035531 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.089047 22580 raft_consensus.cc:979] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: : Instructing follower 2d6860caa69249c1837378d6f5247099 to start an election
I20240501 19:56:17.089167 22580 raft_consensus.cc:1067] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Signalling peer 2d6860caa69249c1837378d6f5247099 to start an election
I20240501 19:56:17.089238 22579 raft_consensus.cc:979] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: : Instructing follower b01a4a9860ce4bff90c3dc414228d44e to start an election
I20240501 19:56:17.089368 22579 raft_consensus.cc:1067] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Signalling peer b01a4a9860ce4bff90c3dc414228d44e to start an election
I20240501 19:56:17.089601 20921 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "afb520db25524bc0b8deee1f443f6a44"
dest_uuid: "2d6860caa69249c1837378d6f5247099"
from {username='slave'} at 127.17.65.2:55009
I20240501 19:56:17.089725 20921 raft_consensus.cc:482] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:17.089789 20921 raft_consensus.cc:3037] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:17.089807 20790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e05bb1fdf29d48b99d3aa029e315a519"
dest_uuid: "b01a4a9860ce4bff90c3dc414228d44e"
from {username='slave'} at 127.17.65.2:46075
I20240501 19:56:17.089917 20790 raft_consensus.cc:482] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:17.089988 20790 raft_consensus.cc:3037] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:17.090883 20790 raft_consensus.cc:504] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:17.090883 20921 raft_consensus.cc:504] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:17.091228 20790 leader_election.cc:288] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [CANDIDATE]: Term 2 election: Requested vote from peers 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:17.091284 20921 leader_election.cc:288] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [CANDIDATE]: Term 2 election: Requested vote from peers 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021)
I20240501 19:56:17.091579 19797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05bb1fdf29d48b99d3aa029e315a519" candidate_uuid: "b01a4a9860ce4bff90c3dc414228d44e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
I20240501 19:56:17.091714 19797 raft_consensus.cc:3032] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:17.091764 19797 raft_consensus.cc:729] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:17.091892 19797 consensus_queue.cc:260] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:17.091928 19796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb520db25524bc0b8deee1f443f6a44" candidate_uuid: "2d6860caa69249c1837378d6f5247099" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1af1fefd33104e7bad976e2bd786c0b6"
I20240501 19:56:17.092031 19796 raft_consensus.cc:3032] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:17.092090 19796 raft_consensus.cc:729] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1af1fefd33104e7bad976e2bd786c0b6, State: Running, Role: LEADER
I20240501 19:56:17.092226 19796 consensus_queue.cc:260] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:17.092037 19797 raft_consensus.cc:3037] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:17.092350 19796 raft_consensus.cc:3037] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:17.093159 19796 raft_consensus.cc:2454] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d6860caa69249c1837378d6f5247099 in term 2.
I20240501 19:56:17.093271 19797 raft_consensus.cc:2454] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01a4a9860ce4bff90c3dc414228d44e in term 2.
I20240501 19:56:17.093501 20854 leader_election.cc:302] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6, 2d6860caa69249c1837378d6f5247099; no voters:
I20240501 19:56:17.093588 20723 leader_election.cc:302] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1af1fefd33104e7bad976e2bd786c0b6, b01a4a9860ce4bff90c3dc414228d44e; no voters:
I20240501 19:56:17.093829 22520 raft_consensus.cc:2781] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:17.093830 22583 raft_consensus.cc:2781] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:17.093983 22520 raft_consensus.cc:686] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 2 LEADER]: Becoming Leader. State: Replica: b01a4a9860ce4bff90c3dc414228d44e, State: Running, Role: LEADER
I20240501 19:56:17.093983 22583 raft_consensus.cc:686] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Becoming Leader. State: Replica: 2d6860caa69249c1837378d6f5247099, State: Running, Role: LEADER
I20240501 19:56:17.094110 22583 consensus_queue.cc:237] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:17.094111 22520 consensus_queue.cc:237] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:17.095798 19517 catalog_manager.cc:5484] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 reported cstate change: term changed from 1 to 2, leader changed from 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) to 2d6860caa69249c1837378d6f5247099 (127.17.65.4). New cstate: current_term: 2 leader_uuid: "2d6860caa69249c1837378d6f5247099" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:17.095798 19524 catalog_manager.cc:5484] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e reported cstate change: term changed from 1 to 2, leader changed from 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) to b01a4a9860ce4bff90c3dc414228d44e (127.17.65.3). New cstate: current_term: 2 leader_uuid: "b01a4a9860ce4bff90c3dc414228d44e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 } } peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:17.161245 19665 raft_consensus.cc:1261] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Refusing update from remote peer 2d6860caa69249c1837378d6f5247099: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:17.161842 22583 consensus_queue.cc:1035] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:17.319994 19645 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:17.322798 21163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:17.324605 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.325688 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.331187 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.334332 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.336881 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.388820 20790 consensus_queue.cc:237] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:17.389550 22520 raft_consensus.cc:2932] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:17.391817 19520 catalog_manager.cc:5484] T ecdb17582af74a12b63acbbe2560166d P b01a4a9860ce4bff90c3dc414228d44e reported cstate change: config changed from index 2 to 4, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "b01a4a9860ce4bff90c3dc414228d44e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:17.394510 19777 tablet_service.cc:1514] Processing DeleteTablet for tablet ecdb17582af74a12b63acbbe2560166d with delete_type TABLET_DATA_TOMBSTONED (TS 1af1fefd33104e7bad976e2bd786c0b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48136
I20240501 19:56:17.394986 22653 tablet_replica.cc:330] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: stopping tablet replica
I20240501 19:56:17.395274 22653 raft_consensus.cc:2229] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:17.395516 22653 raft_consensus.cc:2258] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:17.396530 22653 ts_tablet_manager.cc:1875] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:17.398689 22653 ts_tablet_manager.cc:1888] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 19:56:17.398761 22653 log.cc:1196] T ecdb17582af74a12b63acbbe2560166d P 1af1fefd33104e7bad976e2bd786c0b6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/ecdb17582af74a12b63acbbe2560166d
I20240501 19:56:17.399403 19508 catalog_manager.cc:4830] TS 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021): tablet ecdb17582af74a12b63acbbe2560166d (table TestTable [id=11ff0b9ce562494cad1b0f88aceb3660]) successfully deleted
I20240501 19:56:17.436767 22117 raft_consensus.cc:979] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: : Instructing follower 5bdf040be08b4aa292256860320503f7 to start an election
I20240501 19:56:17.436954 22117 raft_consensus.cc:1067] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Signalling peer 5bdf040be08b4aa292256860320503f7 to start an election
I20240501 19:56:17.437465 21183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "089c2d27c8c2479e8efafa68ebf12969"
dest_uuid: "5bdf040be08b4aa292256860320503f7"
from {username='slave'} at 127.17.65.1:49397
I20240501 19:56:17.437623 21183 raft_consensus.cc:482] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:17.437698 21183 raft_consensus.cc:3037] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:17.438910 21183 raft_consensus.cc:504] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:17.439324 21183 leader_election.cc:288] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [CANDIDATE]: Term 2 election: Requested vote from peers 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:17.439790 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089c2d27c8c2479e8efafa68ebf12969" candidate_uuid: "5bdf040be08b4aa292256860320503f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
I20240501 19:56:17.439911 19665 raft_consensus.cc:3032] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:17.439951 19665 raft_consensus.cc:729] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:17.440071 19665 consensus_queue.cc:260] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:17.440196 19665 raft_consensus.cc:3037] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:17.441112 19665 raft_consensus.cc:2454] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bdf040be08b4aa292256860320503f7 in term 2.
I20240501 19:56:17.441462 21116 leader_election.cc:302] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6, 5bdf040be08b4aa292256860320503f7; no voters:
I20240501 19:56:17.441687 22582 raft_consensus.cc:2781] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:17.441841 22582 raft_consensus.cc:686] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 2 LEADER]: Becoming Leader. State: Replica: 5bdf040be08b4aa292256860320503f7, State: Running, Role: LEADER
I20240501 19:56:17.441960 22582 consensus_queue.cc:237] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:17.443372 19517 catalog_manager.cc:5484] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 reported cstate change: term changed from 1 to 2, leader changed from 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) to 5bdf040be08b4aa292256860320503f7 (127.17.65.6). New cstate: current_term: 2 leader_uuid: "5bdf040be08b4aa292256860320503f7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:17.456040 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.457001 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.458595 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.458681 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:17.459494 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.460060 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.483834 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.519634 21183 consensus_queue.cc:237] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:17.520324 22582 raft_consensus.cc:2932] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:17.522485 19517 catalog_manager.cc:5484] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 5bdf040be08b4aa292256860320503f7 reported cstate change: config changed from index 4 to 6, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "5bdf040be08b4aa292256860320503f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:17.525171 19777 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c3ffbbb4d140bbae02fdb2fed1e109 with delete_type TABLET_DATA_TOMBSTONED (TS 1af1fefd33104e7bad976e2bd786c0b6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48136
I20240501 19:56:17.525558 22653 tablet_replica.cc:330] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: stopping tablet replica
I20240501 19:56:17.525709 22653 raft_consensus.cc:2229] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:17.525828 22653 raft_consensus.cc:2258] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:17.526327 22653 ts_tablet_manager.cc:1875] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:17.528173 22653 ts_tablet_manager.cc:1888] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:17.528239 22653 log.cc:1196] T 32c3ffbbb4d140bbae02fdb2fed1e109 P 1af1fefd33104e7bad976e2bd786c0b6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/32c3ffbbb4d140bbae02fdb2fed1e109
I20240501 19:56:17.528752 19508 catalog_manager.cc:4830] TS 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021): tablet 32c3ffbbb4d140bbae02fdb2fed1e109 (table rebalance_test_table_3 [id=6e9d5ba6b7a740b49a263f77826f6763]) successfully deleted
I20240501 19:56:17.567306 19797 raft_consensus.cc:1261] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Refusing update from remote peer b01a4a9860ce4bff90c3dc414228d44e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:17.567819 22520 consensus_queue.cc:1035] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:17.580608 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:17.582783 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.584149 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.584607 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.585023 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.587808 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.589821 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.611601 19797 raft_consensus.cc:1261] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Refusing update from remote peer 2d6860caa69249c1837378d6f5247099: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:17.612068 22585 consensus_queue.cc:1035] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1fefd33104e7bad976e2bd786c0b6" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:17.641422 21314 consensus_queue.cc:237] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:17.642103 22631 raft_consensus.cc:2932] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:17.644343 19524 catalog_manager.cc:5484] T 618965d7b3d44bf58c00664087392002 P fa2c654d61eb4d32967da7ea2374d738 reported cstate change: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New cstate: current_term: 2 leader_uuid: "fa2c654d61eb4d32967da7ea2374d738" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:17.646747 19645 tablet_service.cc:1514] Processing DeleteTablet for tablet 618965d7b3d44bf58c00664087392002 with delete_type TABLET_DATA_TOMBSTONED (TS 002bd5b7fe4c40169448f30e32ec54c6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60078
I20240501 19:56:17.647192 22768 tablet_replica.cc:330] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: stopping tablet replica
I20240501 19:56:17.648149 22768 raft_consensus.cc:2229] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:17.648309 22768 raft_consensus.cc:2258] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:17.649223 22768 ts_tablet_manager.cc:1875] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:17.651306 22768 ts_tablet_manager.cc:1888] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:17.651408 22768 log.cc:1196] T 618965d7b3d44bf58c00664087392002 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/618965d7b3d44bf58c00664087392002
I20240501 19:56:17.651926 19508 catalog_manager.cc:4830] TS 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451): tablet 618965d7b3d44bf58c00664087392002 (table rebalance_test_table_4 [id=eec65c7f9008472d9417285bc1f2057b]) successfully deleted
I20240501 19:56:17.696729 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:17.698136 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.698694 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.705976 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.706835 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.713465 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.714588 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.768229 21052 consensus_queue.cc:237] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:17.768965 22501 raft_consensus.cc:2932] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:17.770964 19524 catalog_manager.cc:5484] T c57a563e77234a7b99dc890e2d5b2fa9 P fd3145d4045844a99067befa34a077be reported cstate change: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New cstate: current_term: 2 leader_uuid: "fd3145d4045844a99067befa34a077be" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:17.773339 19645 tablet_service.cc:1514] Processing DeleteTablet for tablet c57a563e77234a7b99dc890e2d5b2fa9 with delete_type TABLET_DATA_TOMBSTONED (TS 002bd5b7fe4c40169448f30e32ec54c6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60078
I20240501 19:56:17.773578 22768 tablet_replica.cc:330] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: stopping tablet replica
I20240501 19:56:17.773738 22768 raft_consensus.cc:2229] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:17.773851 22768 raft_consensus.cc:2258] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:17.774451 22768 ts_tablet_manager.cc:1875] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:17.776589 22768 ts_tablet_manager.cc:1888] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:17.776654 22768 log.cc:1196] T c57a563e77234a7b99dc890e2d5b2fa9 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/c57a563e77234a7b99dc890e2d5b2fa9
I20240501 19:56:17.777112 19508 catalog_manager.cc:4830] TS 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451): tablet c57a563e77234a7b99dc890e2d5b2fa9 (table rebalance_test_table_4 [id=eec65c7f9008472d9417285bc1f2057b]) successfully deleted
I20240501 19:56:17.824398 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:17.826947 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.828143 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.829603 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.833835 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.833983 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.855321 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.885607 21314 consensus_queue.cc:237] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } }
I20240501 19:56:17.886287 22631 raft_consensus.cc:2932] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:17.888733 19516 catalog_manager.cc:5484] T 68a0476d7a0c4d1f8dda6182816f4a17 P fa2c654d61eb4d32967da7ea2374d738 reported cstate change: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New cstate: current_term: 2 leader_uuid: "fa2c654d61eb4d32967da7ea2374d738" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fa2c654d61eb4d32967da7ea2374d738" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 34015 } } }
I20240501 19:56:17.891199 19645 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a0476d7a0c4d1f8dda6182816f4a17 with delete_type TABLET_DATA_TOMBSTONED (TS 002bd5b7fe4c40169448f30e32ec54c6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60078
I20240501 19:56:17.891420 22768 tablet_replica.cc:330] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: stopping tablet replica
I20240501 19:56:17.891559 22768 raft_consensus.cc:2229] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:17.891672 22768 raft_consensus.cc:2258] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:17.892158 22768 ts_tablet_manager.cc:1875] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:17.894232 22768 ts_tablet_manager.cc:1888] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:17.894304 22768 log.cc:1196] T 68a0476d7a0c4d1f8dda6182816f4a17 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/68a0476d7a0c4d1f8dda6182816f4a17
I20240501 19:56:17.894814 19508 catalog_manager.cc:4830] TS 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451): tablet 68a0476d7a0c4d1f8dda6182816f4a17 (table rebalance_test_table_2 [id=4a641b337afe4629915afc6453555f8c]) successfully deleted
I20240501 19:56:17.938827 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:17.942006 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.950227 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.951195 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.951962 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.952873 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.952899 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:17.989490 19665 raft_consensus.cc:1261] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Refusing update from remote peer 5bdf040be08b4aa292256860320503f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:17.989900 22582 consensus_queue.cc:1035] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:17.999677 21052 consensus_queue.cc:237] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } }
I20240501 19:56:18.000375 22817 raft_consensus.cc:2932] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:18.002408 19517 catalog_manager.cc:5484] T 8ec069697dd9474fab37c77aa8abdb28 P fd3145d4045844a99067befa34a077be reported cstate change: config changed from index 4 to 6, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "fd3145d4045844a99067befa34a077be" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd3145d4045844a99067befa34a077be" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 41697 } } }
I20240501 19:56:18.004801 19777 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec069697dd9474fab37c77aa8abdb28 with delete_type TABLET_DATA_TOMBSTONED (TS 1af1fefd33104e7bad976e2bd786c0b6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48136
I20240501 19:56:18.005033 22653 tablet_replica.cc:330] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: stopping tablet replica
I20240501 19:56:18.005164 22653 raft_consensus.cc:2229] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:18.005263 22653 raft_consensus.cc:2258] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:18.005757 22653 ts_tablet_manager.cc:1875] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:18.007864 22653 ts_tablet_manager.cc:1888] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:18.007941 22653 log.cc:1196] T 8ec069697dd9474fab37c77aa8abdb28 P 1af1fefd33104e7bad976e2bd786c0b6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ec069697dd9474fab37c77aa8abdb28
I20240501 19:56:18.008428 19508 catalog_manager.cc:4830] TS 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021): tablet 8ec069697dd9474fab37c77aa8abdb28 (table rebalance_test_table_3 [id=6e9d5ba6b7a740b49a263f77826f6763]) successfully deleted
I20240501 19:56:18.057039 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.059226 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:18.067832 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.068430 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.079825 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.080559 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.084959 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.130946 20921 consensus_queue.cc:237] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:18.131562 22583 raft_consensus.cc:2932] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:18.133747 19518 catalog_manager.cc:5484] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 reported cstate change: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New cstate: current_term: 2 leader_uuid: "2d6860caa69249c1837378d6f5247099" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:18.136521 19645 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de3fc3f434240748a8f75260808f9e5 with delete_type TABLET_DATA_TOMBSTONED (TS 002bd5b7fe4c40169448f30e32ec54c6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60078
I20240501 19:56:18.136730 22768 tablet_replica.cc:330] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: stopping tablet replica
I20240501 19:56:18.136866 22768 raft_consensus.cc:2229] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:18.136968 22768 raft_consensus.cc:2258] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:18.137594 22768 ts_tablet_manager.cc:1875] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:18.139539 22768 ts_tablet_manager.cc:1888] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:18.139609 22768 log.cc:1196] T 2de3fc3f434240748a8f75260808f9e5 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/2de3fc3f434240748a8f75260808f9e5
I20240501 19:56:18.140071 19508 catalog_manager.cc:4830] TS 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451): tablet 2de3fc3f434240748a8f75260808f9e5 (table rebalance_test_table_2 [id=4a641b337afe4629915afc6453555f8c]) successfully deleted
I20240501 19:56:18.185896 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.189936 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:18.190474 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.191656 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.197489 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.214718 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.214823 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.252738 20790 consensus_queue.cc:237] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } }
I20240501 19:56:18.253299 22992 raft_consensus.cc:2932] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:18.255494 19518 catalog_manager.cc:5484] T e05bb1fdf29d48b99d3aa029e315a519 P b01a4a9860ce4bff90c3dc414228d44e reported cstate change: config changed from index 4 to 6, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "b01a4a9860ce4bff90c3dc414228d44e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b01a4a9860ce4bff90c3dc414228d44e" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 44125 } } }
I20240501 19:56:18.258235 19777 tablet_service.cc:1514] Processing DeleteTablet for tablet e05bb1fdf29d48b99d3aa029e315a519 with delete_type TABLET_DATA_TOMBSTONED (TS 1af1fefd33104e7bad976e2bd786c0b6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48136
I20240501 19:56:18.258482 22653 tablet_replica.cc:330] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: stopping tablet replica
I20240501 19:56:18.258626 22653 raft_consensus.cc:2229] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:18.258733 22653 raft_consensus.cc:2258] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:18.259366 22653 ts_tablet_manager.cc:1875] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:18.261628 22653 ts_tablet_manager.cc:1888] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:18.261725 22653 log.cc:1196] T e05bb1fdf29d48b99d3aa029e315a519 P 1af1fefd33104e7bad976e2bd786c0b6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/e05bb1fdf29d48b99d3aa029e315a519
I20240501 19:56:18.262177 19508 catalog_manager.cc:4830] TS 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021): tablet e05bb1fdf29d48b99d3aa029e315a519 (table rebalance_test_table_1 [id=06418a4164d9480faaf291fc1c1d0a69]) successfully deleted
I20240501 19:56:18.308301 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.311915 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:18.314054 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.318480 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.318883 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.323757 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.343801 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.374867 20921 consensus_queue.cc:237] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:18.375478 22980 raft_consensus.cc:2932] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:18.377697 19516 catalog_manager.cc:5484] T afb520db25524bc0b8deee1f443f6a44 P 2d6860caa69249c1837378d6f5247099 reported cstate change: config changed from index 4 to 6, VOTER 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "2d6860caa69249c1837378d6f5247099" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:18.380216 19777 tablet_service.cc:1514] Processing DeleteTablet for tablet afb520db25524bc0b8deee1f443f6a44 with delete_type TABLET_DATA_TOMBSTONED (TS 1af1fefd33104e7bad976e2bd786c0b6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48136
I20240501 19:56:18.380424 22653 tablet_replica.cc:330] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: stopping tablet replica
I20240501 19:56:18.380549 22653 raft_consensus.cc:2229] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:18.380657 22653 raft_consensus.cc:2258] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:18.381290 22653 ts_tablet_manager.cc:1875] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:18.383167 22653 ts_tablet_manager.cc:1888] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:18.383234 22653 log.cc:1196] T afb520db25524bc0b8deee1f443f6a44 P 1af1fefd33104e7bad976e2bd786c0b6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/afb520db25524bc0b8deee1f443f6a44
I20240501 19:56:18.383672 19508 catalog_manager.cc:4830] TS 1af1fefd33104e7bad976e2bd786c0b6 (127.17.65.2:42021): tablet afb520db25524bc0b8deee1f443f6a44 (table rebalance_test_table_1 [id=06418a4164d9480faaf291fc1c1d0a69]) successfully deleted
I20240501 19:56:18.435132 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:18.438321 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.438930 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.443562 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.444270 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.446722 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.447504 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:18.501291 21183 consensus_queue.cc:237] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } }
I20240501 19:56:18.502051 23140 raft_consensus.cc:2932] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:18.504298 19517 catalog_manager.cc:5484] T 089c2d27c8c2479e8efafa68ebf12969 P 5bdf040be08b4aa292256860320503f7 reported cstate change: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New cstate: current_term: 2 leader_uuid: "5bdf040be08b4aa292256860320503f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5bdf040be08b4aa292256860320503f7" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 42633 } } }
I20240501 19:56:18.506467 19645 tablet_service.cc:1514] Processing DeleteTablet for tablet 089c2d27c8c2479e8efafa68ebf12969 with delete_type TABLET_DATA_TOMBSTONED (TS 002bd5b7fe4c40169448f30e32ec54c6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60078
I20240501 19:56:18.506719 22768 tablet_replica.cc:330] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: stopping tablet replica
I20240501 19:56:18.506847 22768 raft_consensus.cc:2229] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:18.506970 22768 raft_consensus.cc:2258] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:18.507567 22768 ts_tablet_manager.cc:1875] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:18.509295 22768 ts_tablet_manager.cc:1888] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:18.509366 22768 log.cc:1196] T 089c2d27c8c2479e8efafa68ebf12969 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/089c2d27c8c2479e8efafa68ebf12969
I20240501 19:56:18.509801 19508 catalog_manager.cc:4830] TS 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451): tablet 089c2d27c8c2479e8efafa68ebf12969 (table rebalance_test_table_0 [id=aa25c285e4c541c6b76edca67986e9af]) successfully deleted
I20240501 19:56:18.778401 19665 consensus_queue.cc:237] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
W20240501 19:56:18.779807 19601 consensus_peers.cc:485] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 -> Peer 2d6860caa69249c1837378d6f5247099 (127.17.65.4:42703): Couldn't send request to peer 2d6860caa69249c1837378d6f5247099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 678cb6a860ce43f3a52aab82fa7540a9. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:19.307739 23210 ts_tablet_manager.cc:902] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: Initiating tablet copy from peer 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:19.308215 23210 tablet_copy_client.cc:308] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:36451
I20240501 19:56:19.308727 19686 tablet_copy_service.cc:140] P 002bd5b7fe4c40169448f30e32ec54c6: Received BeginTabletCopySession request for tablet 678cb6a860ce43f3a52aab82fa7540a9 from peer 2d6860caa69249c1837378d6f5247099 ({username='slave'} at 127.17.65.4:38495)
I20240501 19:56:19.308866 19686 tablet_copy_service.cc:161] P 002bd5b7fe4c40169448f30e32ec54c6: Beginning new tablet copy session on tablet 678cb6a860ce43f3a52aab82fa7540a9 from peer 2d6860caa69249c1837378d6f5247099 at {username='slave'} at 127.17.65.4:38495: session id = 2d6860caa69249c1837378d6f5247099-678cb6a860ce43f3a52aab82fa7540a9
I20240501 19:56:19.309633 19686 tablet_copy_source_session.cc:205] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:19.310464 23210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678cb6a860ce43f3a52aab82fa7540a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.312224 23210 tablet_copy_client.cc:783] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:19.312348 23210 tablet_copy_client.cc:647] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:19.313505 23210 tablet_copy_client.cc:523] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:19.314460 23210 tablet_bootstrap.cc:492] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: Bootstrap starting.
I20240501 19:56:19.334697 23210 tablet_bootstrap.cc:492] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:19.335058 23210 tablet_bootstrap.cc:492] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: Bootstrap complete.
I20240501 19:56:19.335253 23210 ts_tablet_manager.cc:1367] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.003s
I20240501 19:56:19.335805 23210 raft_consensus.cc:348] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:19.336033 23210 raft_consensus.cc:729] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6860caa69249c1837378d6f5247099, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.336138 23210 consensus_queue.cc:260] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:19.336531 23210 ts_tablet_manager.cc:1398] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:19.337208 19686 tablet_copy_service.cc:342] P 002bd5b7fe4c40169448f30e32ec54c6: Request end of tablet copy session 2d6860caa69249c1837378d6f5247099-678cb6a860ce43f3a52aab82fa7540a9 received from {username='slave'} at 127.17.65.4:38495
I20240501 19:56:19.337287 19686 tablet_copy_service.cc:433] P 002bd5b7fe4c40169448f30e32ec54c6: ending tablet copy session 2d6860caa69249c1837378d6f5247099-678cb6a860ce43f3a52aab82fa7540a9 on tablet 678cb6a860ce43f3a52aab82fa7540a9 with peer 2d6860caa69249c1837378d6f5247099
I20240501 19:56:19.882561 20921 raft_consensus.cc:1203] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:19.883467 23219 raft_consensus.cc:2932] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:19.883826 20921 raft_consensus.cc:2932] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:19.884707 23219 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:39638 (request call id 0) took 1106 ms. Trace:
I20240501 19:56:19.884775 23219 rpcz_store.cc:276] 0501 19:56:18.778060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:56:18.778122 (+ 62us) service_pool.cc:225] Handling call
0501 19:56:18.778797 (+ 675us) log.cc:842] Serialized 254 byte log entry
0501 19:56:19.884688 (+1105891us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":145,"thread_start_us":98,"threads_started":2,"wal-append.queue_time_us":126,"wal-append.run_cpu_time_us":353,"wal-append.run_wall_time_us":1000414}
I20240501 19:56:19.886380 19517 catalog_manager.cc:5484] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 reported cstate change: config changed from index -1 to 4, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:19.944272 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:19.946062 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:19.951189 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:19.953811 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:19.961192 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:19.961298 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:19.974413 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.000979 19665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "678cb6a860ce43f3a52aab82fa7540a9"
dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39664
I20240501 19:56:20.001119 19665 raft_consensus.cc:595] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:20.051072 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.052102 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.052327 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:20.055115 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.067312 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.074709 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.092743 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.366606 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.368078 19645 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:20.371533 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.379495 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.380302 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.386499 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.392351 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.421959 19665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "678cb6a860ce43f3a52aab82fa7540a9"
dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39708
I20240501 19:56:20.422116 19665 raft_consensus.cc:595] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:20.438874 23357 raft_consensus.cc:979] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: : Instructing follower 2d6860caa69249c1837378d6f5247099 to start an election
I20240501 19:56:20.438989 23357 raft_consensus.cc:1067] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Signalling peer 2d6860caa69249c1837378d6f5247099 to start an election
I20240501 19:56:20.439464 20921 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "678cb6a860ce43f3a52aab82fa7540a9"
dest_uuid: "2d6860caa69249c1837378d6f5247099"
from {username='slave'} at 127.17.65.1:54961
I20240501 19:56:20.439639 20921 raft_consensus.cc:482] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:20.439714 20921 raft_consensus.cc:3037] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:20.440546 20921 raft_consensus.cc:504] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:20.440822 20921 leader_election.cc:288] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [CANDIDATE]: Term 2 election: Requested vote from peers 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:20.441318 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678cb6a860ce43f3a52aab82fa7540a9" candidate_uuid: "2d6860caa69249c1837378d6f5247099" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
I20240501 19:56:20.441438 19665 raft_consensus.cc:3032] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:20.441483 19665 raft_consensus.cc:729] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:20.441632 19665 consensus_queue.cc:260] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:20.441756 19665 raft_consensus.cc:3037] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:20.442487 19665 raft_consensus.cc:2454] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d6860caa69249c1837378d6f5247099 in term 2.
I20240501 19:56:20.442817 20854 leader_election.cc:302] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6, 2d6860caa69249c1837378d6f5247099; no voters:
I20240501 19:56:20.443109 23358 raft_consensus.cc:2781] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:20.443264 23358 raft_consensus.cc:686] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Becoming Leader. State: Replica: 2d6860caa69249c1837378d6f5247099, State: Running, Role: LEADER
I20240501 19:56:20.443441 23358 consensus_queue.cc:237] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:20.445075 19516 catalog_manager.cc:5484] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 reported cstate change: term changed from 1 to 2, leader changed from 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) to 2d6860caa69249c1837378d6f5247099 (127.17.65.4). New cstate: current_term: 2 leader_uuid: "2d6860caa69249c1837378d6f5247099" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:20.479849 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.480775 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:20.482200 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:20.483292 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.495745 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.499068 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.501199 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.782719 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:20.786852 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.788228 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.791188 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:20.792871 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.793859 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.799948 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:20.878878 19665 raft_consensus.cc:1261] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Refusing update from remote peer 2d6860caa69249c1837378d6f5247099: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:20.879395 23359 consensus_queue.cc:1035] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:21.087788 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.087849 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.091823 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.099243 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.099339 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.100775 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.120424 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.151715 20921 consensus_queue.cc:237] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:21.152292 23358 raft_consensus.cc:2932] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:21.154512 19521 catalog_manager.cc:5484] T 678cb6a860ce43f3a52aab82fa7540a9 P 2d6860caa69249c1837378d6f5247099 reported cstate change: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New cstate: current_term: 2 leader_uuid: "2d6860caa69249c1837378d6f5247099" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } }
I20240501 19:56:21.157059 19645 tablet_service.cc:1514] Processing DeleteTablet for tablet 678cb6a860ce43f3a52aab82fa7540a9 with delete_type TABLET_DATA_TOMBSTONED (TS 002bd5b7fe4c40169448f30e32ec54c6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60078
I20240501 19:56:21.157368 23484 tablet_replica.cc:330] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: stopping tablet replica
I20240501 19:56:21.157483 23484 raft_consensus.cc:2229] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:21.157567 23484 raft_consensus.cc:2258] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:21.158093 23484 ts_tablet_manager.cc:1875] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:21.159792 23484 ts_tablet_manager.cc:1888] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:21.159855 23484 log.cc:1196] T 678cb6a860ce43f3a52aab82fa7540a9 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/678cb6a860ce43f3a52aab82fa7540a9
I20240501 19:56:21.160307 19508 catalog_manager.cc:4830] TS 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451): tablet 678cb6a860ce43f3a52aab82fa7540a9 (table rebalance_test_table_0 [id=aa25c285e4c541c6b76edca67986e9af]) successfully deleted
I20240501 19:56:21.414824 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.415563 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.416915 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.418087 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.420343 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.428921 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.440053 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.519260 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.520862 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.522084 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.529265 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.531222 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.537494 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.551177 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.766467 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.766506 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.771693 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.775334 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.776638 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.780748 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.789477 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.902500 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.902921 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.903046 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.908891 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.924433 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.925310 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:21.932899 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.021380 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:22.022204 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.026254 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:22.032552 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.033845 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.034387 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.051793 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
def032d7b005429b87e9fe6941d0e6e6 | 127.17.65.62:37761 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.17.65.20:43447 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
002bd5b7fe4c40169448f30e32ec54c6 | 127.17.65.1:36451 | HEALTHY | <none> | 3 | 0
1af1fefd33104e7bad976e2bd786c0b6 | 127.17.65.2:42021 | HEALTHY | <none> | 2 | 0
2d6860caa69249c1837378d6f5247099 | 127.17.65.4:42703 | HEALTHY | <none> | 3 | 0
5bdf040be08b4aa292256860320503f7 | 127.17.65.6:42633 | HEALTHY | <none> | 2 | 0
b01a4a9860ce4bff90c3dc414228d44e | 127.17.65.3:44125 | HEALTHY | <none> | 2 | 0
fa2c654d61eb4d32967da7ea2374d738 | 127.17.65.7:34015 | HEALTHY | <none> | 2 | 0
fd3145d4045844a99067befa34a077be | 127.17.65.5:41697 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.17.65.1 | experimental | 127.17.65.1:36451
local_ip_for_outbound_sockets | 127.17.65.2 | experimental | 127.17.65.2:42021
local_ip_for_outbound_sockets | 127.17.65.3 | experimental | 127.17.65.3:44125
local_ip_for_outbound_sockets | 127.17.65.4 | experimental | 127.17.65.4:42703
local_ip_for_outbound_sockets | 127.17.65.5 | experimental | 127.17.65.5:41697
local_ip_for_outbound_sockets | 127.17.65.6 | experimental | 127.17.65.6:42633
local_ip_for_outbound_sockets | 127.17.65.7 | experimental | 127.17.65.7:34015
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.17.65.1:36451
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.17.65.2:42021
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.17.65.3:44125
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.17.65.4:42703
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.17.65.5:41697
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.17.65.6:42633
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.17.65.7:34015
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.17.65.20:43447 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:56:22.077980 17668 log_verifier.cc:126] Checking tablet 089c2d27c8c2479e8efafa68ebf12969
I20240501 19:56:22.088975 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 089c2d27c8c2479e8efafa68ebf12969
I20240501 19:56:22.089073 17668 log_verifier.cc:126] Checking tablet 251b0a3550a44aa7b266d253b479a5b6
I20240501 19:56:22.097254 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 251b0a3550a44aa7b266d253b479a5b6
I20240501 19:56:22.097314 17668 log_verifier.cc:126] Checking tablet 2de3fc3f434240748a8f75260808f9e5
I20240501 19:56:22.105639 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2de3fc3f434240748a8f75260808f9e5
I20240501 19:56:22.105698 17668 log_verifier.cc:126] Checking tablet 32c3ffbbb4d140bbae02fdb2fed1e109
I20240501 19:56:22.114137 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 32c3ffbbb4d140bbae02fdb2fed1e109
I20240501 19:56:22.114200 17668 log_verifier.cc:126] Checking tablet 618965d7b3d44bf58c00664087392002
I20240501 19:56:22.122541 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 618965d7b3d44bf58c00664087392002
I20240501 19:56:22.122601 17668 log_verifier.cc:126] Checking tablet 678cb6a860ce43f3a52aab82fa7540a9
I20240501 19:56:22.131143 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 678cb6a860ce43f3a52aab82fa7540a9
I20240501 19:56:22.131199 17668 log_verifier.cc:126] Checking tablet 68a0476d7a0c4d1f8dda6182816f4a17
I20240501 19:56:22.139684 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 68a0476d7a0c4d1f8dda6182816f4a17
I20240501 19:56:22.139753 17668 log_verifier.cc:126] Checking tablet 6a3b72769a4c41349d9052c8e740f009
I20240501 19:56:22.147953 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6a3b72769a4c41349d9052c8e740f009
I20240501 19:56:22.148012 17668 log_verifier.cc:126] Checking tablet 6fae90ed925f41999fe254adcf950841
I20240501 19:56:22.156152 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6fae90ed925f41999fe254adcf950841
I20240501 19:56:22.156214 17668 log_verifier.cc:126] Checking tablet 8c8e23f52a6e46aeae2d71e1ed04e9a7
I20240501 19:56:22.165112 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8c8e23f52a6e46aeae2d71e1ed04e9a7
I20240501 19:56:22.165172 17668 log_verifier.cc:126] Checking tablet 8ec069697dd9474fab37c77aa8abdb28
I20240501 19:56:22.174044 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8ec069697dd9474fab37c77aa8abdb28
I20240501 19:56:22.174114 17668 log_verifier.cc:126] Checking tablet 97866b7451304498ab8e4300f6853e7b
I20240501 19:56:22.182571 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 97866b7451304498ab8e4300f6853e7b
I20240501 19:56:22.182631 17668 log_verifier.cc:126] Checking tablet afb520db25524bc0b8deee1f443f6a44
I20240501 19:56:22.191224 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet afb520db25524bc0b8deee1f443f6a44
I20240501 19:56:22.191289 17668 log_verifier.cc:126] Checking tablet c57a563e77234a7b99dc890e2d5b2fa9
I20240501 19:56:22.199738 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c57a563e77234a7b99dc890e2d5b2fa9
I20240501 19:56:22.199793 17668 log_verifier.cc:126] Checking tablet e05bb1fdf29d48b99d3aa029e315a519
I20240501 19:56:22.208335 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e05bb1fdf29d48b99d3aa029e315a519
I20240501 19:56:22.208391 17668 log_verifier.cc:126] Checking tablet ecdb17582af74a12b63acbbe2560166d
I20240501 19:56:22.216043 17668 log_verifier.cc:177] Verified matching terms for 4 ops in tablet ecdb17582af74a12b63acbbe2560166d
I20240501 19:56:22.218106 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.8:0
--local_ip_for_outbound_sockets=127.17.65.8
--webserver_interface=127.17.65.8
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--builtin_ntp_servers=127.17.65.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:56:22.329910 23660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:22.330149 23660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:22.330193 23660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:22.333318 23660 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:22.333397 23660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:22.333467 23660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.8
I20240501 19:56:22.337273 23660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.17.65.8
--webserver_port=0
--tserver_master_addrs=127.17.65.62:37761
--never_fsync=true
--heap_profile_path=/tmp/kudu.23660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.8
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:22.338250 23660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:22.339468 23660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:22.347677 23665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:22.348361 23666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:22.349335 23668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:22.350190 23660 server_base.cc:1034] running on GCE node
I20240501 19:56:22.350551 23660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:22.351111 23660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:22.352259 23660 hybrid_clock.cc:648] HybridClock initialized: now 1714593382352242 us; error 30 us; skew 500 ppm
I20240501 19:56:22.353967 23660 webserver.cc:460] Webserver started at http://127.17.65.8:46445/ using document root <none> and password file <none>
I20240501 19:56:22.354470 23660 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:22.354532 23660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:22.354692 23660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:22.356261 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "446fa3b84a074abc9af77c270244ebbc"
format_stamp: "Formatted at 2024-05-01 19:56:22 on dist-test-slave-703j"
I20240501 19:56:22.356694 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "446fa3b84a074abc9af77c270244ebbc"
format_stamp: "Formatted at 2024-05-01 19:56:22 on dist-test-slave-703j"
I20240501 19:56:22.359761 23660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 19:56:22.361704 23674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.362602 23660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:22.362695 23660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "446fa3b84a074abc9af77c270244ebbc"
format_stamp: "Formatted at 2024-05-01 19:56:22 on dist-test-slave-703j"
I20240501 19:56:22.362769 23660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:22.372675 23660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:22.373193 23660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:22.373334 23660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:22.373790 23660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:22.374570 23660 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:22.374632 23660 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.374670 23660 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:22.374691 23660 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.382496 23660 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.8:36681
I20240501 19:56:22.382516 23787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.8:36681 every 8 connection(s)
I20240501 19:56:22.383421 23660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:22.391885 23788 heartbeater.cc:347] Connected to a master server at 127.17.65.62:37761
I20240501 19:56:22.392096 23788 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:22.392505 23788 heartbeater.cc:511] Master 127.17.65.62:37761 requested a full tablet report, sending...
I20240501 19:56:22.393361 19521 ts_manager.cc:194] Registered new tserver with Master: 446fa3b84a074abc9af77c270244ebbc (127.17.65.8:36681)
I20240501 19:56:22.393523 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 23660
I20240501 19:56:22.393643 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:56:22.393959 19521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.8:49763
I20240501 19:56:22.580793 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.581924 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:22.582337 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.584604 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:22.596009 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.603283 23722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:22.604234 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.608050 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.735994 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:22.740137 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.740533 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:22.748229 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.753587 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.765588 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.766294 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:22.773200 23722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:22.812114 19665 consensus_queue.cc:237] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
W20240501 19:56:22.816329 19600 consensus_peers.cc:485] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 -> Peer 446fa3b84a074abc9af77c270244ebbc (127.17.65.8:36681): Couldn't send request to peer 446fa3b84a074abc9af77c270244ebbc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fae90ed925f41999fe254adcf950841. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:23.277472 23893 ts_tablet_manager.cc:902] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: Initiating tablet copy from peer 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:23.278142 23893 tablet_copy_client.cc:308] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:36451
I20240501 19:56:23.282276 19686 tablet_copy_service.cc:140] P 002bd5b7fe4c40169448f30e32ec54c6: Received BeginTabletCopySession request for tablet 6fae90ed925f41999fe254adcf950841 from peer 446fa3b84a074abc9af77c270244ebbc ({username='slave'} at 127.17.65.8:43395)
I20240501 19:56:23.282389 19686 tablet_copy_service.cc:161] P 002bd5b7fe4c40169448f30e32ec54c6: Beginning new tablet copy session on tablet 6fae90ed925f41999fe254adcf950841 from peer 446fa3b84a074abc9af77c270244ebbc at {username='slave'} at 127.17.65.8:43395: session id = 446fa3b84a074abc9af77c270244ebbc-6fae90ed925f41999fe254adcf950841
I20240501 19:56:23.283366 19686 tablet_copy_source_session.cc:205] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:23.286499 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fae90ed925f41999fe254adcf950841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.291024 23893 tablet_copy_client.cc:783] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:23.291250 23893 tablet_copy_client.cc:647] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:23.292658 23893 tablet_copy_client.cc:523] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:23.294035 23893 tablet_bootstrap.cc:492] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: Bootstrap starting.
I20240501 19:56:23.299189 23893 log.cc:824] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:23.318917 23893 tablet_bootstrap.cc:492] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:23.319331 23893 tablet_bootstrap.cc:492] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: Bootstrap complete.
I20240501 19:56:23.319648 23893 ts_tablet_manager.cc:1367] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240501 19:56:23.321528 23893 raft_consensus.cc:348] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } }
I20240501 19:56:23.322067 23893 raft_consensus.cc:729] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446fa3b84a074abc9af77c270244ebbc, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.322332 23893 consensus_queue.cc:260] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:23.322918 23788 heartbeater.cc:503] Master 127.17.65.62:37761 was elected leader, sending a full tablet report...
I20240501 19:56:23.323118 23893 ts_tablet_manager.cc:1398] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:23.323837 19686 tablet_copy_service.cc:342] P 002bd5b7fe4c40169448f30e32ec54c6: Request end of tablet copy session 446fa3b84a074abc9af77c270244ebbc-6fae90ed925f41999fe254adcf950841 received from {username='slave'} at 127.17.65.8:43395
I20240501 19:56:23.323927 19686 tablet_copy_service.cc:433] P 002bd5b7fe4c40169448f30e32ec54c6: ending tablet copy session 446fa3b84a074abc9af77c270244ebbc-6fae90ed925f41999fe254adcf950841 on tablet 6fae90ed925f41999fe254adcf950841 with peer 446fa3b84a074abc9af77c270244ebbc
I20240501 19:56:23.887290 23742 raft_consensus.cc:1203] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:23.888458 23903 raft_consensus.cc:2932] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 446fa3b84a074abc9af77c270244ebbc (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:23.889011 23742 raft_consensus.cc:2932] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 446fa3b84a074abc9af77c270244ebbc (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:23.890033 23903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:39820 (request call id 0) took 1078 ms. Trace:
I20240501 19:56:23.890175 23903 rpcz_store.cc:276] 0501 19:56:22.811799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:56:22.811849 (+ 50us) service_pool.cc:225] Handling call
0501 19:56:22.812548 (+ 699us) log.cc:842] Serialized 254 byte log entry
0501 19:56:23.890006 (+1077458us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":56,"raft.queue_time_us":104,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":93,"threads_started":2,"wal-append.queue_time_us":113,"wal-append.run_cpu_time_us":286,"wal-append.run_wall_time_us":1000346}
I20240501 19:56:23.891788 19521 catalog_manager.cc:5484] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc reported cstate change: config changed from index -1 to 4, VOTER 446fa3b84a074abc9af77c270244ebbc (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "002bd5b7fe4c40169448f30e32ec54c6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:23.908296 20921 consensus_queue.cc:237] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
W20240501 19:56:23.912564 20855 consensus_peers.cc:485] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 -> Peer 446fa3b84a074abc9af77c270244ebbc (127.17.65.8:36681): Couldn't send request to peer 446fa3b84a074abc9af77c270244ebbc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2de3fc3f434240748a8f75260808f9e5. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.425928 23932 ts_tablet_manager.cc:902] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: Initiating tablet copy from peer 2d6860caa69249c1837378d6f5247099 (127.17.65.4:42703)
I20240501 19:56:24.426412 23932 tablet_copy_client.cc:308] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:42703
I20240501 19:56:24.429896 20941 tablet_copy_service.cc:140] P 2d6860caa69249c1837378d6f5247099: Received BeginTabletCopySession request for tablet 2de3fc3f434240748a8f75260808f9e5 from peer 446fa3b84a074abc9af77c270244ebbc ({username='slave'} at 127.17.65.8:41793)
I20240501 19:56:24.430006 20941 tablet_copy_service.cc:161] P 2d6860caa69249c1837378d6f5247099: Beginning new tablet copy session on tablet 2de3fc3f434240748a8f75260808f9e5 from peer 446fa3b84a074abc9af77c270244ebbc at {username='slave'} at 127.17.65.8:41793: session id = 446fa3b84a074abc9af77c270244ebbc-2de3fc3f434240748a8f75260808f9e5
I20240501 19:56:24.431257 20941 tablet_copy_source_session.cc:205] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:24.432413 23932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de3fc3f434240748a8f75260808f9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.434617 23932 tablet_copy_client.cc:783] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:24.434757 23932 tablet_copy_client.cc:647] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:24.436017 23932 tablet_copy_client.cc:523] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:24.436941 23932 tablet_bootstrap.cc:492] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: Bootstrap starting.
I20240501 19:56:24.454385 23932 tablet_bootstrap.cc:492] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:24.454659 23932 tablet_bootstrap.cc:492] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: Bootstrap complete.
I20240501 19:56:24.454806 23932 ts_tablet_manager.cc:1367] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240501 19:56:24.455287 23932 raft_consensus.cc:348] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } }
I20240501 19:56:24.455489 23932 raft_consensus.cc:729] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446fa3b84a074abc9af77c270244ebbc, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.455601 23932 consensus_queue.cc:260] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:24.455909 23932 ts_tablet_manager.cc:1398] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 19:56:24.456516 20941 tablet_copy_service.cc:342] P 2d6860caa69249c1837378d6f5247099: Request end of tablet copy session 446fa3b84a074abc9af77c270244ebbc-2de3fc3f434240748a8f75260808f9e5 received from {username='slave'} at 127.17.65.8:41793
I20240501 19:56:24.456609 20941 tablet_copy_service.cc:433] P 2d6860caa69249c1837378d6f5247099: ending tablet copy session 446fa3b84a074abc9af77c270244ebbc-2de3fc3f434240748a8f75260808f9e5 on tablet 2de3fc3f434240748a8f75260808f9e5 with peer 446fa3b84a074abc9af77c270244ebbc
I20240501 19:56:24.859236 23742 raft_consensus.cc:1203] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.6->[2.7-2.7] Dedup: 2.7->[]
I20240501 19:56:24.860183 23942 raft_consensus.cc:2932] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 446fa3b84a074abc9af77c270244ebbc (127.17.65.8) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:24.860533 23742 raft_consensus.cc:2932] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 446fa3b84a074abc9af77c270244ebbc (127.17.65.8) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:24.863265 19517 catalog_manager.cc:5484] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc reported cstate change: config changed from index 6 to 7, VOTER 446fa3b84a074abc9af77c270244ebbc (127.17.65.8) added. New cstate: current_term: 2 leader_uuid: "2d6860caa69249c1837378d6f5247099" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:24.922135 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:24.927419 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:24.928401 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:24.928617 23722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:24.934311 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:24.935282 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:24.935554 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:24.947517 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:24.979149 20921 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2de3fc3f434240748a8f75260808f9e5"
dest_uuid: "2d6860caa69249c1837378d6f5247099"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52060
I20240501 19:56:24.979322 20921 raft_consensus.cc:595] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Received request to transfer leadership
I20240501 19:56:25.029004 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.029598 23722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:25.031440 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:25.033779 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.041574 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.053165 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.059295 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:25.060225 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.139462 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:25.140951 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:25.149041 23722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:25.149441 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.150743 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.156922 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.164829 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.185122 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.205490 19665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6fae90ed925f41999fe254adcf950841"
dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35344
I20240501 19:56:25.205636 19665 raft_consensus.cc:595] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:25.258966 19645 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:25.259229 20901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:25.260707 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.262909 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.262934 23722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:25.269273 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.289542 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.301355 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.307089 23931 raft_consensus.cc:979] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: : Instructing follower 446fa3b84a074abc9af77c270244ebbc to start an election
I20240501 19:56:25.307201 23931 raft_consensus.cc:1067] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Signalling peer 446fa3b84a074abc9af77c270244ebbc to start an election
I20240501 19:56:25.307775 23742 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2de3fc3f434240748a8f75260808f9e5"
dest_uuid: "446fa3b84a074abc9af77c270244ebbc"
from {username='slave'} at 127.17.65.4:56127
I20240501 19:56:25.307981 23742 raft_consensus.cc:482] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:25.308075 23742 raft_consensus.cc:3037] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:25.309171 23742 raft_consensus.cc:504] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:25.310092 23742 leader_election.cc:288] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [CANDIDATE]: Term 3 election: Requested vote from peers 2d6860caa69249c1837378d6f5247099 (127.17.65.4:42703)
I20240501 19:56:25.310492 20921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de3fc3f434240748a8f75260808f9e5" candidate_uuid: "446fa3b84a074abc9af77c270244ebbc" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "2d6860caa69249c1837378d6f5247099"
I20240501 19:56:25.310644 20921 raft_consensus.cc:3032] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:56:25.310689 20921 raft_consensus.cc:729] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d6860caa69249c1837378d6f5247099, State: Running, Role: LEADER
I20240501 19:56:25.310838 20921 consensus_queue.cc:260] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:25.310976 20921 raft_consensus.cc:3037] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:25.311805 20921 raft_consensus.cc:2454] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446fa3b84a074abc9af77c270244ebbc in term 3.
I20240501 19:56:25.312160 23677 leader_election.cc:302] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2d6860caa69249c1837378d6f5247099, 446fa3b84a074abc9af77c270244ebbc; no voters:
I20240501 19:56:25.312422 23941 raft_consensus.cc:2781] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:56:25.312628 23941 raft_consensus.cc:686] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 3 LEADER]: Becoming Leader. State: Replica: 446fa3b84a074abc9af77c270244ebbc, State: Running, Role: LEADER
I20240501 19:56:25.312934 23941 consensus_queue.cc:237] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:25.315287 19518 catalog_manager.cc:5484] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc reported cstate change: term changed from 2 to 3, leader changed from 2d6860caa69249c1837378d6f5247099 (127.17.65.4) to 446fa3b84a074abc9af77c270244ebbc (127.17.65.8). New cstate: current_term: 3 leader_uuid: "446fa3b84a074abc9af77c270244ebbc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:25.540328 24122 raft_consensus.cc:979] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: : Instructing follower 446fa3b84a074abc9af77c270244ebbc to start an election
I20240501 19:56:25.540452 24122 raft_consensus.cc:1067] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Signalling peer 446fa3b84a074abc9af77c270244ebbc to start an election
I20240501 19:56:25.540896 23742 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6fae90ed925f41999fe254adcf950841"
dest_uuid: "446fa3b84a074abc9af77c270244ebbc"
from {username='slave'} at 127.17.65.1:34817
I20240501 19:56:25.541041 23742 raft_consensus.cc:482] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:25.541090 23742 raft_consensus.cc:3037] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.542095 23742 raft_consensus.cc:504] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:25.542471 23742 leader_election.cc:288] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [CANDIDATE]: Term 2 election: Requested vote from peers 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451)
I20240501 19:56:25.542968 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fae90ed925f41999fe254adcf950841" candidate_uuid: "446fa3b84a074abc9af77c270244ebbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "002bd5b7fe4c40169448f30e32ec54c6"
I20240501 19:56:25.543172 19665 raft_consensus.cc:3032] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:25.543221 19665 raft_consensus.cc:729] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 002bd5b7fe4c40169448f30e32ec54c6, State: Running, Role: LEADER
I20240501 19:56:25.543354 19665 consensus_queue.cc:260] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:25.543483 19665 raft_consensus.cc:3037] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.544462 19665 raft_consensus.cc:2454] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446fa3b84a074abc9af77c270244ebbc in term 2.
I20240501 19:56:25.544778 23675 leader_election.cc:302] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 002bd5b7fe4c40169448f30e32ec54c6, 446fa3b84a074abc9af77c270244ebbc; no voters:
I20240501 19:56:25.544996 23941 raft_consensus.cc:2781] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:25.545121 23941 raft_consensus.cc:686] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 2 LEADER]: Becoming Leader. State: Replica: 446fa3b84a074abc9af77c270244ebbc, State: Running, Role: LEADER
I20240501 19:56:25.545250 23941 consensus_queue.cc:237] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:25.546624 19518 catalog_manager.cc:5484] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc reported cstate change: term changed from 1 to 2, leader changed from 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) to 446fa3b84a074abc9af77c270244ebbc (127.17.65.8). New cstate: current_term: 2 leader_uuid: "446fa3b84a074abc9af77c270244ebbc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 } } peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:25.585070 19645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.589505 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.590159 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.591778 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.599507 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.599988 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.618723 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.623214 23722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.697913 19645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.698176 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.698664 23722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.698940 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.701750 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.705277 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.708627 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.728196 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:25.851361 20921 raft_consensus.cc:1261] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 3 FOLLOWER]: Refusing update from remote peer 446fa3b84a074abc9af77c270244ebbc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 19:56:25.851907 23941 consensus_queue.cc:1035] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d6860caa69249c1837378d6f5247099" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 42703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:26.005709 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.012984 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.014042 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.015094 19645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.015414 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.016100 23722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.035249 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.037212 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.060092 19665 raft_consensus.cc:1261] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Refusing update from remote peer 446fa3b84a074abc9af77c270244ebbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:26.060798 24109 consensus_queue.cc:1035] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "002bd5b7fe4c40169448f30e32ec54c6" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:26.089071 23742 consensus_queue.cc:237] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:26.089725 23941 raft_consensus.cc:2932] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) evicted. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:26.092090 19517 catalog_manager.cc:5484] T 2de3fc3f434240748a8f75260808f9e5 P 446fa3b84a074abc9af77c270244ebbc reported cstate change: config changed from index 7 to 9, VOTER 2d6860caa69249c1837378d6f5247099 (127.17.65.4) evicted. New cstate: current_term: 3 leader_uuid: "446fa3b84a074abc9af77c270244ebbc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:26.098587 20901 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de3fc3f434240748a8f75260808f9e5 with delete_type TABLET_DATA_TOMBSTONED (TS 2d6860caa69249c1837378d6f5247099 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52100
I20240501 19:56:26.099017 24257 tablet_replica.cc:330] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: stopping tablet replica
I20240501 19:56:26.099372 24257 raft_consensus.cc:2229] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:26.099558 24257 raft_consensus.cc:2258] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:26.101441 24257 ts_tablet_manager.cc:1875] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:26.103545 24257 ts_tablet_manager.cc:1888] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.8
I20240501 19:56:26.103642 24257 log.cc:1196] T 2de3fc3f434240748a8f75260808f9e5 P 2d6860caa69249c1837378d6f5247099: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/2de3fc3f434240748a8f75260808f9e5
I20240501 19:56:26.104197 19510 catalog_manager.cc:4830] TS 2d6860caa69249c1837378d6f5247099 (127.17.65.4:42703): tablet 2de3fc3f434240748a8f75260808f9e5 (table rebalance_test_table_2 [id=4a641b337afe4629915afc6453555f8c]) successfully deleted
I20240501 19:56:26.151253 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.151775 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.151914 19645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.157253 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.158325 23722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.158358 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.168421 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.171308 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.222127 23742 consensus_queue.cc:237] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } }
I20240501 19:56:26.222750 23941 raft_consensus.cc:2932] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:26.224844 19516 catalog_manager.cc:5484] T 6fae90ed925f41999fe254adcf950841 P 446fa3b84a074abc9af77c270244ebbc reported cstate change: config changed from index 4 to 6, VOTER 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1) evicted. New cstate: current_term: 2 leader_uuid: "446fa3b84a074abc9af77c270244ebbc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "446fa3b84a074abc9af77c270244ebbc" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 36681 } } }
I20240501 19:56:26.227312 19645 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fae90ed925f41999fe254adcf950841 with delete_type TABLET_DATA_TOMBSTONED (TS 002bd5b7fe4c40169448f30e32ec54c6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60078
I20240501 19:56:26.227690 24306 tablet_replica.cc:330] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: stopping tablet replica
I20240501 19:56:26.227820 24306 raft_consensus.cc:2229] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:26.227936 24306 raft_consensus.cc:2258] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:26.229333 24306 ts_tablet_manager.cc:1875] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:26.231691 24306 ts_tablet_manager.cc:1888] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:26.231774 24306 log.cc:1196] T 6fae90ed925f41999fe254adcf950841 P 002bd5b7fe4c40169448f30e32ec54c6: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/6fae90ed925f41999fe254adcf950841
I20240501 19:56:26.232270 19508 catalog_manager.cc:4830] TS 002bd5b7fe4c40169448f30e32ec54c6 (127.17.65.1:36451): tablet 6fae90ed925f41999fe254adcf950841 (table rebalance_test_table_4 [id=eec65c7f9008472d9417285bc1f2057b]) successfully deleted
I20240501 19:56:26.497800 19645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.503841 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.504942 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.505925 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.506798 23722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.515929 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.531906 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.541072 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.613760 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.617542 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.620024 19645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.622793 23722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.639185 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.641002 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.642599 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.657019 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.735149 20901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.741169 20770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.745728 21294 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.746196 21032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.749217 23722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.755717 19645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.758519 19777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:26.764106 21163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
def032d7b005429b87e9fe6941d0e6e6 | 127.17.65.62:37761 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.17.65.20:43447 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
002bd5b7fe4c40169448f30e32ec54c6 | 127.17.65.1:36451 | HEALTHY | <none> | 2 | 0
1af1fefd33104e7bad976e2bd786c0b6 | 127.17.65.2:42021 | HEALTHY | <none> | 2 | 0
2d6860caa69249c1837378d6f5247099 | 127.17.65.4:42703 | HEALTHY | <none> | 2 | 0
446fa3b84a074abc9af77c270244ebbc | 127.17.65.8:36681 | HEALTHY | <none> | 2 | 0
5bdf040be08b4aa292256860320503f7 | 127.17.65.6:42633 | HEALTHY | <none> | 2 | 0
b01a4a9860ce4bff90c3dc414228d44e | 127.17.65.3:44125 | HEALTHY | <none> | 2 | 0
fa2c654d61eb4d32967da7ea2374d738 | 127.17.65.7:34015 | HEALTHY | <none> | 2 | 0
fd3145d4045844a99067befa34a077be | 127.17.65.5:41697 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.17.65.1 | experimental | 127.17.65.1:36451
local_ip_for_outbound_sockets | 127.17.65.2 | experimental | 127.17.65.2:42021
local_ip_for_outbound_sockets | 127.17.65.3 | experimental | 127.17.65.3:44125
local_ip_for_outbound_sockets | 127.17.65.4 | experimental | 127.17.65.4:42703
local_ip_for_outbound_sockets | 127.17.65.5 | experimental | 127.17.65.5:41697
local_ip_for_outbound_sockets | 127.17.65.6 | experimental | 127.17.65.6:42633
local_ip_for_outbound_sockets | 127.17.65.7 | experimental | 127.17.65.7:34015
local_ip_for_outbound_sockets | 127.17.65.8 | experimental | 127.17.65.8:36681
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.17.65.1:36451
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.17.65.2:42021
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.17.65.3:44125
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.17.65.4:42703
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.17.65.5:41697
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.17.65.6:42633
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.17.65.7:34015
server_dump_info_path | /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.17.65.8:36681
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.17.65.20:43447 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:56:26.793145 17668 log_verifier.cc:126] Checking tablet 089c2d27c8c2479e8efafa68ebf12969
I20240501 19:56:26.799356 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 089c2d27c8c2479e8efafa68ebf12969
I20240501 19:56:26.799417 17668 log_verifier.cc:126] Checking tablet 251b0a3550a44aa7b266d253b479a5b6
I20240501 19:56:26.804699 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 251b0a3550a44aa7b266d253b479a5b6
I20240501 19:56:26.804751 17668 log_verifier.cc:126] Checking tablet 2de3fc3f434240748a8f75260808f9e5
I20240501 19:56:26.813095 17668 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 2de3fc3f434240748a8f75260808f9e5
I20240501 19:56:26.813161 17668 log_verifier.cc:126] Checking tablet 32c3ffbbb4d140bbae02fdb2fed1e109
I20240501 19:56:26.818714 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 32c3ffbbb4d140bbae02fdb2fed1e109
I20240501 19:56:26.818771 17668 log_verifier.cc:126] Checking tablet 618965d7b3d44bf58c00664087392002
I20240501 19:56:26.824355 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 618965d7b3d44bf58c00664087392002
I20240501 19:56:26.824415 17668 log_verifier.cc:126] Checking tablet 678cb6a860ce43f3a52aab82fa7540a9
I20240501 19:56:26.830219 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 678cb6a860ce43f3a52aab82fa7540a9
I20240501 19:56:26.830274 17668 log_verifier.cc:126] Checking tablet 68a0476d7a0c4d1f8dda6182816f4a17
I20240501 19:56:26.835938 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 68a0476d7a0c4d1f8dda6182816f4a17
I20240501 19:56:26.835992 17668 log_verifier.cc:126] Checking tablet 6a3b72769a4c41349d9052c8e740f009
I20240501 19:56:26.841357 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6a3b72769a4c41349d9052c8e740f009
I20240501 19:56:26.841413 17668 log_verifier.cc:126] Checking tablet 6fae90ed925f41999fe254adcf950841
I20240501 19:56:26.849726 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6fae90ed925f41999fe254adcf950841
I20240501 19:56:26.849800 17668 log_verifier.cc:126] Checking tablet 8c8e23f52a6e46aeae2d71e1ed04e9a7
I20240501 19:56:26.855386 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8c8e23f52a6e46aeae2d71e1ed04e9a7
I20240501 19:56:26.855443 17668 log_verifier.cc:126] Checking tablet 8ec069697dd9474fab37c77aa8abdb28
I20240501 19:56:26.861138 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8ec069697dd9474fab37c77aa8abdb28
I20240501 19:56:26.861196 17668 log_verifier.cc:126] Checking tablet 97866b7451304498ab8e4300f6853e7b
I20240501 19:56:26.866994 17668 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 97866b7451304498ab8e4300f6853e7b
I20240501 19:56:26.867084 17668 log_verifier.cc:126] Checking tablet afb520db25524bc0b8deee1f443f6a44
I20240501 19:56:26.872875 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet afb520db25524bc0b8deee1f443f6a44
I20240501 19:56:26.872932 17668 log_verifier.cc:126] Checking tablet c57a563e77234a7b99dc890e2d5b2fa9
I20240501 19:56:26.878629 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c57a563e77234a7b99dc890e2d5b2fa9
I20240501 19:56:26.878681 17668 log_verifier.cc:126] Checking tablet e05bb1fdf29d48b99d3aa029e315a519
I20240501 19:56:26.884104 17668 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e05bb1fdf29d48b99d3aa029e315a519
I20240501 19:56:26.884153 17668 log_verifier.cc:126] Checking tablet ecdb17582af74a12b63acbbe2560166d
I20240501 19:56:26.888756 17668 log_verifier.cc:177] Verified matching terms for 4 ops in tablet ecdb17582af74a12b63acbbe2560166d
I20240501 19:56:26.889178 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 19584
I20240501 19:56:26.896198 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 19715
I20240501 19:56:26.903198 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 20708
I20240501 19:56:26.908663 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 20839
I20240501 19:56:26.914547 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 20970
I20240501 19:56:26.919787 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 21100
I20240501 19:56:26.924928 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 21232
I20240501 19:56:26.929920 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 23660
I20240501 19:56:26.935357 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 19493
2024-05-01T19:56:26Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (30077 ms)
[----------] 1 test from RebalanceParamTest (30077 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240501 19:56:26.974236 17668 test_util.cc:268] Using random seed: -507261931
I20240501 19:56:26.974995 17668 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:56:26.975054 17668 ts_itest-base.cc:113] --------------
I20240501 19:56:26.975072 17668 ts_itest-base.cc:114] 8 tablet servers
I20240501 19:56:26.975090 17668 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:56:26.975106 17668 ts_itest-base.cc:116] --------------
2024-05-01T19:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:26Z Disabled control of system clock
I20240501 19:56:26.983007 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:40811
--webserver_interface=127.17.65.62
--webserver_port=0
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.65.62:40811
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 19:56:27.086735 24423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:27.086983 24423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:27.087051 24423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:27.090045 24423 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:27.090122 24423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:27.090148 24423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:27.090169 24423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:27.090189 24423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:27.093657 24423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.17.65.62:40811
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:40811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.65.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:27.094489 24423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:27.095597 24423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:27.103199 24431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.104056 24429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.104749 24428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:27.105423 24423 server_base.cc:1034] running on GCE node
I20240501 19:56:27.105870 24423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:27.106721 24423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:27.107864 24423 hybrid_clock.cc:648] HybridClock initialized: now 1714593387107840 us; error 33 us; skew 500 ppm
I20240501 19:56:27.109601 24423 webserver.cc:460] Webserver started at http://127.17.65.62:41659/ using document root <none> and password file <none>
I20240501 19:56:27.110106 24423 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:27.110157 24423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:27.110338 24423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:27.111877 24423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "547d05e083494037ac958fdcd041e13f"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.112308 24423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "547d05e083494037ac958fdcd041e13f"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.115352 24423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:27.117360 24437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.118257 24423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:27.118366 24423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
uuid: "547d05e083494037ac958fdcd041e13f"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.118439 24423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:27.128729 24423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:27.129261 24423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:27.129417 24423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:27.135452 24423 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.62:40811
I20240501 19:56:27.135468 24489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.62:40811 every 8 connection(s)
I20240501 19:56:27.136296 24423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:56:27.137915 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 24423
I20240501 19:56:27.138038 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:56:27.138753 24490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.144707 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Bootstrap starting.
I20240501 19:56:27.146950 24490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.147753 24490 log.cc:824] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:27.149183 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: No bootstrap required, opened a new log
I20240501 19:56:27.151708 24490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547d05e083494037ac958fdcd041e13f" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 40811 } }
I20240501 19:56:27.151867 24490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.151916 24490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547d05e083494037ac958fdcd041e13f, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.152225 24490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547d05e083494037ac958fdcd041e13f" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 40811 } }
I20240501 19:56:27.152316 24490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:27.152359 24490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:27.152436 24490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.153041 24490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547d05e083494037ac958fdcd041e13f" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 40811 } }
I20240501 19:56:27.153304 24490 leader_election.cc:302] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547d05e083494037ac958fdcd041e13f; no voters:
I20240501 19:56:27.153520 24490 leader_election.cc:288] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:27.153869 24495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:27.154042 24495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [term 1 LEADER]: Becoming Leader. State: Replica: 547d05e083494037ac958fdcd041e13f, State: Running, Role: LEADER
I20240501 19:56:27.154280 24495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547d05e083494037ac958fdcd041e13f" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 40811 } }
I20240501 19:56:27.154491 24490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:27.155709 24497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 547d05e083494037ac958fdcd041e13f. Latest consensus state: current_term: 1 leader_uuid: "547d05e083494037ac958fdcd041e13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547d05e083494037ac958fdcd041e13f" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 40811 } } }
I20240501 19:56:27.155819 24497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:27.156119 24496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "547d05e083494037ac958fdcd041e13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547d05e083494037ac958fdcd041e13f" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 40811 } } }
I20240501 19:56:27.156190 24506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:27.156204 24496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:27.158732 24506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:27.162822 24506 catalog_manager.cc:1347] Generated new cluster ID: 3123cfa824c344b6a1a41de641044e86
I20240501 19:56:27.162881 24506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:27.186862 24506 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:27.187826 24506 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:27.202394 24506 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Generated new TSK 0
I20240501 19:56:27.203068 24506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:27.214010 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.1:0
--local_ip_for_outbound_sockets=127.17.65.1
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:56:27.325613 24514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:27.325829 24514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:27.325888 24514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:27.329957 24514 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:27.330054 24514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:27.330199 24514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.1
I20240501 19:56:27.334978 24514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.24514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.1
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:27.335995 24514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:27.337081 24514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:27.345383 24519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.346314 24522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.347090 24520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:27.347836 24514 server_base.cc:1034] running on GCE node
I20240501 19:56:27.348173 24514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:27.348675 24514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:27.349838 24514 hybrid_clock.cc:648] HybridClock initialized: now 1714593387349822 us; error 28 us; skew 500 ppm
I20240501 19:56:27.351641 24514 webserver.cc:460] Webserver started at http://127.17.65.1:43757/ using document root <none> and password file <none>
I20240501 19:56:27.352209 24514 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:27.352269 24514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:27.352432 24514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:27.353971 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bbf7735f82ca4f65a5170ac7909faa47"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.354423 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bbf7735f82ca4f65a5170ac7909faa47"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.357558 24514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:27.360601 24528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.361506 24514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:27.361608 24514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bbf7735f82ca4f65a5170ac7909faa47"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.361681 24514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:27.377547 24514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:27.378137 24514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:27.378294 24514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:27.378825 24514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:27.379722 24514 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:27.379772 24514 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.379835 24514 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:27.379863 24514 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.388526 24514 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.1:43399
I20240501 19:56:27.388571 24641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.1:43399 every 8 connection(s)
I20240501 19:56:27.389415 24514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:56:27.390285 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 24514
I20240501 19:56:27.390419 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:56:27.392589 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.2:0
--local_ip_for_outbound_sockets=127.17.65.2
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:27.398805 24642 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:27.399015 24642 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:27.399466 24642 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:27.400753 24454 ts_manager.cc:194] Registered new tserver with Master: bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:27.401825 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.1:37569
W20240501 19:56:27.495644 24645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:27.495936 24645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:27.496008 24645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:27.498973 24645 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:27.499070 24645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:27.499179 24645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.2
I20240501 19:56:27.502532 24645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.24645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.2
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:27.503389 24645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:27.504420 24645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:27.512737 24650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.513311 24653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.514302 24651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:27.515180 24645 server_base.cc:1034] running on GCE node
I20240501 19:56:27.515561 24645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:27.516095 24645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:27.517238 24645 hybrid_clock.cc:648] HybridClock initialized: now 1714593387517222 us; error 32 us; skew 500 ppm
I20240501 19:56:27.519023 24645 webserver.cc:460] Webserver started at http://127.17.65.2:43331/ using document root <none> and password file <none>
I20240501 19:56:27.519577 24645 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:27.519646 24645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:27.519807 24645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:27.521397 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9081292597be4a72a42b216fddcd7971"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.521869 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9081292597be4a72a42b216fddcd7971"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.524996 24645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:56:27.526935 24659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.527874 24645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:27.527969 24645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9081292597be4a72a42b216fddcd7971"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.528038 24645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:27.541862 24645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:27.542356 24645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:27.542496 24645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:27.542945 24645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:27.543785 24645 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:27.543833 24645 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.543890 24645 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:27.543921 24645 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.551836 24645 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.2:36299
I20240501 19:56:27.551872 24772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.2:36299 every 8 connection(s)
I20240501 19:56:27.552673 24645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:56:27.558094 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 24645
I20240501 19:56:27.558243 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:56:27.560376 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.3:0
--local_ip_for_outbound_sockets=127.17.65.3
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:27.560664 24773 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:27.560843 24773 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:27.561204 24773 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:27.562120 24454 ts_manager.cc:194] Registered new tserver with Master: 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:27.562626 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.2:40687
W20240501 19:56:27.662864 24776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:27.663110 24776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:27.663182 24776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:27.666137 24776 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:27.666198 24776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:27.666309 24776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.3
I20240501 19:56:27.669816 24776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.24776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.3
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:27.670629 24776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:27.671720 24776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:27.680449 24784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.681035 24781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.681974 24782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:27.682824 24776 server_base.cc:1034] running on GCE node
I20240501 19:56:27.683218 24776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:27.683737 24776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:27.684877 24776 hybrid_clock.cc:648] HybridClock initialized: now 1714593387684863 us; error 28 us; skew 500 ppm
I20240501 19:56:27.686566 24776 webserver.cc:460] Webserver started at http://127.17.65.3:42039/ using document root <none> and password file <none>
I20240501 19:56:27.687110 24776 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:27.687171 24776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:27.687345 24776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:27.688871 24776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9d64cdfaeb2d46b29845160e395fc241"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.689311 24776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9d64cdfaeb2d46b29845160e395fc241"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.692390 24776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 19:56:27.694267 24790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.695173 24776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:27.695266 24776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9d64cdfaeb2d46b29845160e395fc241"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.695343 24776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:27.733846 24776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:27.734462 24776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:27.734622 24776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:27.735172 24776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:27.736023 24776 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:27.736070 24776 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.736126 24776 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:27.736156 24776 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.744021 24776 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.3:38893
I20240501 19:56:27.744077 24903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.3:38893 every 8 connection(s)
I20240501 19:56:27.744948 24776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:27.746385 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 24776
I20240501 19:56:27.746517 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:56:27.748656 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.4:0
--local_ip_for_outbound_sockets=127.17.65.4
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:27.753906 24904 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:27.754117 24904 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:27.754531 24904 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:27.755564 24454 ts_manager.cc:194] Registered new tserver with Master: 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:27.756426 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.3:59303
W20240501 19:56:27.851981 24907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:27.852172 24907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:27.852208 24907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:27.854934 24907 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:27.855000 24907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:27.855126 24907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.4
I20240501 19:56:27.858546 24907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.24907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.4
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:27.859320 24907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:27.860373 24907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:27.868676 24915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.869570 24912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:27.870344 24913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:27.871106 24907 server_base.cc:1034] running on GCE node
I20240501 19:56:27.871419 24907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:27.871888 24907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:27.873029 24907 hybrid_clock.cc:648] HybridClock initialized: now 1714593387873016 us; error 26 us; skew 500 ppm
I20240501 19:56:27.874778 24907 webserver.cc:460] Webserver started at http://127.17.65.4:39859/ using document root <none> and password file <none>
I20240501 19:56:27.875375 24907 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:27.875428 24907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:27.875598 24907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:27.877172 24907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.877594 24907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.880697 24907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:56:27.882714 24921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.883705 24907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:27.883803 24907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
format_stamp: "Formatted at 2024-05-01 19:56:27 on dist-test-slave-703j"
I20240501 19:56:27.883880 24907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:27.911861 24907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:27.912407 24907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:27.912530 24907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:27.912971 24907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:27.913774 24907 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:27.913817 24907 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.913856 24907 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:27.913877 24907 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:27.921451 24907 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.4:45261
I20240501 19:56:27.921511 25034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.4:45261 every 8 connection(s)
I20240501 19:56:27.922300 24907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:27.924566 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 24907
I20240501 19:56:27.924717 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:27.926827 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.5:0
--local_ip_for_outbound_sockets=127.17.65.5
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:27.931406 25035 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:27.931598 25035 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:27.931984 25035 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:27.932899 24454 ts_manager.cc:194] Registered new tserver with Master: b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:27.933427 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.4:43489
W20240501 19:56:28.028510 25038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:28.028723 25038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:28.028784 25038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:28.031749 25038 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:28.031816 25038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:28.031903 25038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.5
I20240501 19:56:28.035511 25038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.25038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.5
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:28.036360 25038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:28.037429 25038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:28.046191 25043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:28.047137 25046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:28.047807 25044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:28.048496 25038 server_base.cc:1034] running on GCE node
I20240501 19:56:28.048868 25038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:28.049379 25038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:28.050535 25038 hybrid_clock.cc:648] HybridClock initialized: now 1714593388050520 us; error 28 us; skew 500 ppm
I20240501 19:56:28.052356 25038 webserver.cc:460] Webserver started at http://127.17.65.5:46813/ using document root <none> and password file <none>
I20240501 19:56:28.052879 25038 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:28.052938 25038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:28.053095 25038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:28.054754 25038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.055228 25038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.058380 25038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:28.060427 25052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.061318 25038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:28.061425 25038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.061502 25038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:28.073009 25038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:28.073498 25038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:28.073644 25038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:28.074118 25038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:28.074923 25038 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:28.075007 25038 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.075090 25038 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:28.075126 25038 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.083384 25038 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.5:45247
I20240501 19:56:28.083403 25165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.5:45247 every 8 connection(s)
I20240501 19:56:28.084259 25038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:28.092306 25166 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:28.092492 25166 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:28.092856 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 25038
I20240501 19:56:28.092880 25166 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:28.092972 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:28.093823 24454 ts_manager.cc:194] Registered new tserver with Master: 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:28.094453 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.5:53097
I20240501 19:56:28.095284 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.6:0
--local_ip_for_outbound_sockets=127.17.65.6
--webserver_interface=127.17.65.6
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:56:28.198436 25169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:28.198647 25169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:28.198705 25169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:28.201779 25169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:28.201840 25169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:28.201917 25169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.6
I20240501 19:56:28.205502 25169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.65.6
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.25169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.6
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:28.206372 25169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:28.207536 25169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:28.217003 25177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:28.217902 25174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:28.219390 25175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:28.219350 25169 server_base.cc:1034] running on GCE node
I20240501 19:56:28.219795 25169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:28.220391 25169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:28.221554 25169 hybrid_clock.cc:648] HybridClock initialized: now 1714593388221537 us; error 39 us; skew 500 ppm
I20240501 19:56:28.223610 25169 webserver.cc:460] Webserver started at http://127.17.65.6:43645/ using document root <none> and password file <none>
I20240501 19:56:28.224154 25169 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:28.224210 25169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:28.224401 25169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:28.226044 25169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "29d1a658e2b946e0b412e5fef82db9d9"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.226547 25169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "29d1a658e2b946e0b412e5fef82db9d9"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.229849 25169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:28.232445 25183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.233381 25169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:28.233484 25169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "29d1a658e2b946e0b412e5fef82db9d9"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.233562 25169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:28.257426 25169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:28.258034 25169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:28.258213 25169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:28.258766 25169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:28.259719 25169 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:28.259769 25169 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.259831 25169 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:28.259857 25169 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.268388 25169 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.6:44579
I20240501 19:56:28.268410 25296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.6:44579 every 8 connection(s)
I20240501 19:56:28.269351 25169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:28.270972 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 25169
I20240501 19:56:28.271126 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:56:28.273424 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.7:0
--local_ip_for_outbound_sockets=127.17.65.7
--webserver_interface=127.17.65.7
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:28.279537 25297 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:28.279768 25297 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:28.280236 25297 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:28.281180 24454 ts_manager.cc:194] Registered new tserver with Master: 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:28.282003 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.6:55469
W20240501 19:56:28.383303 25300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:28.383548 25300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:28.383595 25300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:28.386603 25300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:28.386683 25300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:28.386765 25300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.7
I20240501 19:56:28.390662 25300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.65.7
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.25300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.7
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:28.391731 25300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:28.392949 25300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:28.402159 25306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:28.402784 25305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:28.403471 25308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:28.403839 24642 heartbeater.cc:503] Master 127.17.65.62:40811 was elected leader, sending a full tablet report...
I20240501 19:56:28.404497 25300 server_base.cc:1034] running on GCE node
I20240501 19:56:28.404955 25300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:28.405624 25300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:28.406776 25300 hybrid_clock.cc:648] HybridClock initialized: now 1714593388406750 us; error 40 us; skew 500 ppm
I20240501 19:56:28.408808 25300 webserver.cc:460] Webserver started at http://127.17.65.7:42451/ using document root <none> and password file <none>
I20240501 19:56:28.409399 25300 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:28.409462 25300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:28.409653 25300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:28.411446 25300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "53e34bb09d8e41129a5683079ff63610"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.411947 25300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "53e34bb09d8e41129a5683079ff63610"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.415629 25300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:28.418004 25314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.419108 25300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:28.419220 25300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "53e34bb09d8e41129a5683079ff63610"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.419312 25300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:28.434993 25300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:28.435689 25300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:28.435858 25300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:28.436383 25300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:28.437312 25300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:28.437361 25300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.437417 25300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:28.437448 25300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.445776 25300 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.7:39797
I20240501 19:56:28.445858 25427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.7:39797 every 8 connection(s)
I20240501 19:56:28.446944 25300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:28.450240 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 25300
I20240501 19:56:28.450392 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:56:28.452484 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.8:0
--local_ip_for_outbound_sockets=127.17.65.8
--webserver_interface=127.17.65.8
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:28.456941 25428 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:28.457162 25428 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:28.457617 25428 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:28.458575 24454 ts_manager.cc:194] Registered new tserver with Master: 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:28.459175 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.7:43065
W20240501 19:56:28.563338 25431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:28.563632 25431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:28.563719 25431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 19:56:28.564314 24773 heartbeater.cc:503] Master 127.17.65.62:40811 was elected leader, sending a full tablet report...
W20240501 19:56:28.567253 25431 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:28.567330 25431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:28.567425 25431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.8
I20240501 19:56:28.571152 25431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.17.65.8
--webserver_port=0
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.25431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.8
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:28.572077 25431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:28.573227 25431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:28.582428 25436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:28.583333 25437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:28.584064 25439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:28.584811 25431 server_base.cc:1034] running on GCE node
I20240501 19:56:28.585191 25431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:28.585775 25431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:28.586927 25431 hybrid_clock.cc:648] HybridClock initialized: now 1714593388586909 us; error 28 us; skew 500 ppm
I20240501 19:56:28.589032 25431 webserver.cc:460] Webserver started at http://127.17.65.8:34553/ using document root <none> and password file <none>
I20240501 19:56:28.589586 25431 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:28.589641 25431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:28.589802 25431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:28.591490 25431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4a6e54e86a9949e5970b42e8e32e7561"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.591990 25431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4a6e54e86a9949e5970b42e8e32e7561"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.595222 25431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:28.597394 25445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.598491 25431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:28.598594 25431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4a6e54e86a9949e5970b42e8e32e7561"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:28.598695 25431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:28.626994 25431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:28.627642 25431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:28.627779 25431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:28.628253 25431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:28.629165 25431 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:28.629212 25431 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.629251 25431 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:28.629271 25431 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:28.637485 25431 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.8:35957
I20240501 19:56:28.637513 25558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.8:35957 every 8 connection(s)
I20240501 19:56:28.638404 25431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:28.639273 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 25431
I20240501 19:56:28.639384 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:56:28.646771 25559 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:28.646973 25559 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:28.647493 25559 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:28.648336 24454 ts_manager.cc:194] Registered new tserver with Master: 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:28.648866 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.8:40081
I20240501 19:56:28.653288 17668 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:56:28.662027 17668 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:56:28.662096 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 25038
I20240501 19:56:28.667687 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 25169
I20240501 19:56:28.672760 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 25300
I20240501 19:56:28.677894 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 25431
I20240501 19:56:28.758239 24904 heartbeater.cc:503] Master 127.17.65.62:40811 was elected leader, sending a full tablet report...
I20240501 19:56:28.935248 25035 heartbeater.cc:503] Master 127.17.65.62:40811 was elected leader, sending a full tablet report...
I20240501 19:56:32.434429 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:32.455484 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2964308d1543f081c2c02ca51eb0e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.455016 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2964308d1543f081c2c02ca51eb0e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.455276 24705 tablet_service.cc:1467] Processing CreateTablet for tablet e8dde944a715448eac832f3e74e45138 (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.455072 24837 tablet_service.cc:1467] Processing CreateTablet for tablet a4f17d0a3d0b49beaa435f32bc50530f (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.455351 24706 tablet_service.cc:1467] Processing CreateTablet for tablet a4f17d0a3d0b49beaa435f32bc50530f (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.456229 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2964308d1543f081c2c02ca51eb0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.456252 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2964308d1543f081c2c02ca51eb0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.456547 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f17d0a3d0b49beaa435f32bc50530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.456431 24968 tablet_service.cc:1467] Processing CreateTablet for tablet e8dde944a715448eac832f3e74e45138 (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.457212 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dde944a715448eac832f3e74e45138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.457396 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dde944a715448eac832f3e74e45138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.455015 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2964308d1543f081c2c02ca51eb0e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.459668 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2964308d1543f081c2c02ca51eb0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.460047 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f17d0a3d0b49beaa435f32bc50530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.463640 24576 tablet_service.cc:1467] Processing CreateTablet for tablet a4f17d0a3d0b49beaa435f32bc50530f (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.464216 24575 tablet_service.cc:1467] Processing CreateTablet for tablet e8dde944a715448eac832f3e74e45138 (DEFAULT_TABLE table=rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:32.464632 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f17d0a3d0b49beaa435f32bc50530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.465030 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dde944a715448eac832f3e74e45138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:32.468120 25580 tablet_bootstrap.cc:492] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:32.469125 25581 tablet_bootstrap.cc:492] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:32.469940 25579 tablet_bootstrap.cc:492] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:32.470592 25580 tablet_bootstrap.cc:654] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.471096 25581 tablet_bootstrap.cc:654] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.471747 25580 log.cc:824] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:32.471843 25581 log.cc:824] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:32.472060 25582 tablet_bootstrap.cc:492] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:32.472290 25579 tablet_bootstrap.cc:654] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.473047 25579 log.cc:824] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:32.473532 25581 tablet_bootstrap.cc:492] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:32.473570 25580 tablet_bootstrap.cc:492] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:32.473738 25580 ts_tablet_manager.cc:1367] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:32.473739 25581 ts_tablet_manager.cc:1367] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20240501 19:56:32.474191 25582 tablet_bootstrap.cc:654] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.475100 25582 log.cc:824] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:32.476876 25582 tablet_bootstrap.cc:492] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:32.476779 25580 raft_consensus.cc:348] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.476954 25580 raft_consensus.cc:374] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.477020 25580 raft_consensus.cc:729] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.477056 25582 ts_tablet_manager.cc:1367] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:56:32.477420 25580 consensus_queue.cc:260] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.477617 25581 raft_consensus.cc:348] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.477820 25581 raft_consensus.cc:374] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.477877 25581 raft_consensus.cc:729] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.478243 25580 ts_tablet_manager.cc:1398] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:32.478312 25581 consensus_queue.cc:260] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.478533 25580 tablet_bootstrap.cc:492] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:32.479180 25581 ts_tablet_manager.cc:1398] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240501 19:56:32.479521 25581 tablet_bootstrap.cc:492] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:32.480480 25581 tablet_bootstrap.cc:654] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.480914 25582 raft_consensus.cc:348] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:32.481768 25581 tablet_bootstrap.cc:492] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:32.481117 25582 raft_consensus.cc:374] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.481858 25581 ts_tablet_manager.cc:1367] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:32.481899 25582 raft_consensus.cc:729] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.482362 25582 consensus_queue.cc:260] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:32.482398 25581 raft_consensus.cc:348] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.482501 25581 raft_consensus.cc:374] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.482542 25581 raft_consensus.cc:729] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.482700 25581 consensus_queue.cc:260] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.482952 25581 ts_tablet_manager.cc:1398] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:32.483361 25582 ts_tablet_manager.cc:1398] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 19:56:32.483572 25580 tablet_bootstrap.cc:654] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.483707 25582 tablet_bootstrap.cc:492] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:32.484930 25582 tablet_bootstrap.cc:654] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.485570 25579 tablet_bootstrap.cc:492] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:32.485775 25579 ts_tablet_manager.cc:1367] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240501 19:56:32.486160 25582 tablet_bootstrap.cc:492] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:32.486239 25582 ts_tablet_manager.cc:1367] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:32.486848 25582 raft_consensus.cc:348] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.486956 25582 raft_consensus.cc:374] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.486995 25582 raft_consensus.cc:729] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.487121 25582 consensus_queue.cc:260] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.487447 25582 ts_tablet_manager.cc:1398] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:32.487668 25580 tablet_bootstrap.cc:492] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:32.487762 25580 ts_tablet_manager.cc:1367] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:32.488430 25580 raft_consensus.cc:348] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:32.488585 25580 raft_consensus.cc:374] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.488627 25580 raft_consensus.cc:729] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.488754 25580 consensus_queue.cc:260] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:32.489018 25580 ts_tablet_manager.cc:1398] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:32.489193 25580 tablet_bootstrap.cc:492] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:32.490144 25580 tablet_bootstrap.cc:654] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.491187 25579 raft_consensus.cc:348] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:32.491449 25579 raft_consensus.cc:374] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.491542 25579 raft_consensus.cc:729] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.492009 25580 tablet_bootstrap.cc:492] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:32.492005 25579 consensus_queue.cc:260] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:32.492096 25580 ts_tablet_manager.cc:1367] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:32.492451 25580 raft_consensus.cc:348] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.492537 25580 raft_consensus.cc:374] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.492573 25580 raft_consensus.cc:729] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.492707 25580 consensus_queue.cc:260] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.492966 25580 ts_tablet_manager.cc:1398] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:32.493497 25579 ts_tablet_manager.cc:1398] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 19:56:32.493758 25579 tablet_bootstrap.cc:492] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:32.494815 25579 tablet_bootstrap.cc:654] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.496029 25579 tablet_bootstrap.cc:492] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:32.496114 25579 ts_tablet_manager.cc:1367] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:32.496537 25579 raft_consensus.cc:348] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.496630 25579 raft_consensus.cc:374] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.496659 25579 raft_consensus.cc:729] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.496752 25579 consensus_queue.cc:260] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.496984 25579 ts_tablet_manager.cc:1398] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 19:56:32.499691 24905 tablet.cc:2369] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:32.556320 25587 raft_consensus.cc:482] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:32.556538 25587 raft_consensus.cc:504] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
W20240501 19:56:32.557101 24774 tablet.cc:2369] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:32.557832 25587 leader_election.cc:288] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:32.562309 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dde944a715448eac832f3e74e45138" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:32.562341 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dde944a715448eac832f3e74e45138" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:32.562597 24989 raft_consensus.cc:2454] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:32.562608 24596 raft_consensus.cc:2454] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:32.563007 24661 leader_election.cc:302] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:32.563266 25587 raft_consensus.cc:2781] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:32.563340 25587 raft_consensus.cc:482] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:32.563382 25587 raft_consensus.cc:3037] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.564244 25587 raft_consensus.cc:504] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:32.564574 25587 leader_election.cc:288] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:32.564898 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dde944a715448eac832f3e74e45138" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:32.564975 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dde944a715448eac832f3e74e45138" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:32.565016 24989 raft_consensus.cc:3037] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.565114 24596 raft_consensus.cc:3037] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.565891 24989 raft_consensus.cc:2454] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:32.566028 24596 raft_consensus.cc:2454] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:32.566164 24661 leader_election.cc:302] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:32.566331 25587 raft_consensus.cc:2781] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:32.566553 25587 raft_consensus.cc:686] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:32.566855 25587 consensus_queue.cc:237] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:32.569818 24453 catalog_manager.cc:5484] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:32.572783 25588 raft_consensus.cc:482] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:32.572932 25588 raft_consensus.cc:504] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.573951 25588 leader_election.cc:288] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:32.577569 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2964308d1543f081c2c02ca51eb0e1" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:32.577728 24989 raft_consensus.cc:2454] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:32.577719 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2964308d1543f081c2c02ca51eb0e1" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:32.577893 24727 raft_consensus.cc:2454] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:32.578132 24792 leader_election.cc:302] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:32.578426 25588 raft_consensus.cc:2781] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:32.578495 25588 raft_consensus.cc:482] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:32.578523 25588 raft_consensus.cc:3037] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.579445 25588 raft_consensus.cc:504] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.579795 25588 leader_election.cc:288] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:32.580152 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2964308d1543f081c2c02ca51eb0e1" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:32.580160 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2964308d1543f081c2c02ca51eb0e1" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:32.580257 24727 raft_consensus.cc:3037] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.580269 24989 raft_consensus.cc:3037] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.581059 24989 raft_consensus.cc:2454] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:32.581063 24727 raft_consensus.cc:2454] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:32.581362 24791 leader_election.cc:302] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:32.581552 25588 raft_consensus.cc:2781] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:32.581775 25588 raft_consensus.cc:686] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:32.582075 25588 consensus_queue.cc:237] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.584918 24454 catalog_manager.cc:5484] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
W20240501 19:56:32.644516 24643 tablet.cc:2369] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:32.676779 25036 tablet.cc:2369] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:32.833304 25588 raft_consensus.cc:482] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:32.833452 25588 raft_consensus.cc:504] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.833953 25588 leader_election.cc:288] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:32.834790 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f17d0a3d0b49beaa435f32bc50530f" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:32.834959 24727 raft_consensus.cc:2454] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:32.835276 24791 leader_election.cc:302] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:32.835470 25588 raft_consensus.cc:2781] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:32.835523 25588 raft_consensus.cc:482] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:32.835553 25588 raft_consensus.cc:3037] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.836539 25588 raft_consensus.cc:504] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.836894 25588 leader_election.cc:288] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:32.837286 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f17d0a3d0b49beaa435f32bc50530f" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:32.837393 24727 raft_consensus.cc:3037] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.837759 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f17d0a3d0b49beaa435f32bc50530f" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:32.837879 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f17d0a3d0b49beaa435f32bc50530f" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:32.837894 24596 raft_consensus.cc:2454] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:32.838259 24727 raft_consensus.cc:2454] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:32.838517 24791 leader_election.cc:302] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters: bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:56:32.838676 25588 raft_consensus.cc:2781] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:32.838785 25588 raft_consensus.cc:686] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:32.838929 25588 consensus_queue.cc:237] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:32.840746 24454 catalog_manager.cc:5484] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:32.961571 25588 consensus_queue.cc:1035] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:32.965633 25602 consensus_queue.cc:1035] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:33.010329 25605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:33.010627 25605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:33.013710 25605 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:56:33.115880 25587 consensus_queue.cc:1035] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.118036 24596 raft_consensus.cc:3037] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.118085 24727 raft_consensus.cc:1261] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.118741 25602 consensus_queue.cc:1035] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.119174 24596 raft_consensus.cc:1261] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.120211 25587 consensus_queue.cc:1035] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.121896 25588 consensus_queue.cc:1035] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.129762 25608 mvcc.cc:205] Tried to move back new op lower bound from 7022974538236182528 to 7022974535956344832. Current Snapshot: MvccSnapshot[applied={T|T < 7022974538236182528}]
I20240501 19:56:33.147177 25607 mvcc.cc:205] Tried to move back new op lower bound from 7022974538206932992 to 7022974537069793280. Current Snapshot: MvccSnapshot[applied={T|T < 7022974538190614528}]
I20240501 19:56:33.153702 25645 mvcc.cc:205] Tried to move back new op lower bound from 7022974538206932992 to 7022974537069793280. Current Snapshot: MvccSnapshot[applied={T|T < 7022974538190614528}]
INSERT report
rows total: 2000
time total: 122.134 ms
time per row: 0.0610672 ms
I20240501 19:56:33.185128 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:33.190403 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 5a37dfc7a4324e2b8c43acb51d07a277 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.190464 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 5a37dfc7a4324e2b8c43acb51d07a277 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.190757 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37dfc7a4324e2b8c43acb51d07a277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.190762 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37dfc7a4324e2b8c43acb51d07a277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.191344 24707 tablet_service.cc:1467] Processing CreateTablet for tablet fcfa6701a958416ca481e0349071caa0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.191617 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfa6701a958416ca481e0349071caa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.192776 25671 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:33.193437 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 2e30faedce3742e0ba83de21059613d1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.193686 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e30faedce3742e0ba83de21059613d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.193646 24575 tablet_service.cc:1467] Processing CreateTablet for tablet fcfa6701a958416ca481e0349071caa0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.193874 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfa6701a958416ca481e0349071caa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.194023 25671 tablet_bootstrap.cc:654] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.194306 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 5a37dfc7a4324e2b8c43acb51d07a277 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.194603 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37dfc7a4324e2b8c43acb51d07a277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.195295 25671 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:33.195384 25671 ts_tablet_manager.cc:1367] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:33.195816 25673 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:33.195832 25671 raft_consensus.cc:348] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.195933 25671 raft_consensus.cc:374] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.195968 25671 raft_consensus.cc:729] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.196111 25671 consensus_queue.cc:260] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.196357 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 2e30faedce3742e0ba83de21059613d1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.196584 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e30faedce3742e0ba83de21059613d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.196712 25673 tablet_bootstrap.cc:654] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.197016 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 2e30faedce3742e0ba83de21059613d1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.197244 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e30faedce3742e0ba83de21059613d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.197836 25673 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:33.197923 25673 ts_tablet_manager.cc:1367] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:33.196367 25671 ts_tablet_manager.cc:1398] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:33.198259 25671 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:33.198390 25673 raft_consensus.cc:348] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.198515 25673 raft_consensus.cc:374] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.198552 25673 raft_consensus.cc:729] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.198683 25673 consensus_queue.cc:260] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.199085 25673 ts_tablet_manager.cc:1398] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:33.199156 25671 tablet_bootstrap.cc:654] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.199319 24836 tablet_service.cc:1467] Processing CreateTablet for tablet fcfa6701a958416ca481e0349071caa0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.199537 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfa6701a958416ca481e0349071caa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.200742 25671 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:33.200876 25671 ts_tablet_manager.cc:1367] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:33.200975 25672 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:33.201396 25671 raft_consensus.cc:348] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.201566 25671 raft_consensus.cc:374] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.201642 25671 raft_consensus.cc:729] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.201865 25671 consensus_queue.cc:260] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.202203 25671 ts_tablet_manager.cc:1398] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:33.202391 25675 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:33.202623 25671 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:33.202944 25672 tablet_bootstrap.cc:654] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.202935 25644 raft_consensus.cc:482] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.203023 25644 raft_consensus.cc:504] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.203513 25644 leader_election.cc:288] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:33.203802 25675 tablet_bootstrap.cc:654] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.204133 25671 tablet_bootstrap.cc:654] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.204311 25672 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:33.204414 25672 ts_tablet_manager.cc:1367] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:33.204824 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfa6701a958416ca481e0349071caa0" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:33.204957 24596 raft_consensus.cc:2454] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:33.204932 25672 raft_consensus.cc:348] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.205047 25672 raft_consensus.cc:374] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.205071 25675 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:33.205086 25672 raft_consensus.cc:729] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.205150 25675 ts_tablet_manager.cc:1367] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:33.205209 25672 consensus_queue.cc:260] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.205288 24660 leader_election.cc:302] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:33.205443 25644 raft_consensus.cc:2781] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.205498 25644 raft_consensus.cc:482] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.205505 25672 ts_tablet_manager.cc:1398] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.205546 25644 raft_consensus.cc:3037] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.205693 25672 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:33.205629 25675 raft_consensus.cc:348] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.205734 25675 raft_consensus.cc:374] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.205765 25675 raft_consensus.cc:729] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.206609 25672 tablet_bootstrap.cc:654] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.206585 25644 raft_consensus.cc:504] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.206928 25644 leader_election.cc:288] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:33.207248 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfa6701a958416ca481e0349071caa0" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:33.207381 24596 raft_consensus.cc:3037] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.207485 25675 consensus_queue.cc:260] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.207785 25675 ts_tablet_manager.cc:1398] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:33.208408 24596 raft_consensus.cc:2454] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:33.208554 25675 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:33.208675 24660 leader_election.cc:302] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:33.209466 25644 raft_consensus.cc:2781] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.209563 25644 raft_consensus.cc:686] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:33.209745 25671 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:33.209720 25644 consensus_queue.cc:237] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.209939 25671 ts_tablet_manager.cc:1367] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:33.210423 25671 raft_consensus.cc:348] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.210541 25671 raft_consensus.cc:374] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.210579 25671 raft_consensus.cc:729] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.210605 25675 tablet_bootstrap.cc:654] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.210731 25671 consensus_queue.cc:260] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.210976 25671 ts_tablet_manager.cc:1398] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:33.211448 24451 catalog_manager.cc:5484] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:33.212920 25675 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:33.212997 25675 ts_tablet_manager.cc:1367] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:56:33.213402 25675 raft_consensus.cc:348] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.214404 25675 raft_consensus.cc:374] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.214361 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfa6701a958416ca481e0349071caa0" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:33.214450 25675 raft_consensus.cc:729] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.214592 25675 consensus_queue.cc:260] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.214905 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfa6701a958416ca481e0349071caa0" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:33.215150 25672 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:33.215241 25672 ts_tablet_manager.cc:1367] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
W20240501 19:56:33.215296 24661 leader_election.cc:341] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:33.215569 25675 ts_tablet_manager.cc:1398] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
W20240501 19:56:33.215590 24661 leader_election.cc:341] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:33.215721 25675 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:33.215752 25672 raft_consensus.cc:348] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.215857 25672 raft_consensus.cc:374] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.215895 25672 raft_consensus.cc:729] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.216013 25672 consensus_queue.cc:260] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.216310 25672 ts_tablet_manager.cc:1398] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.216764 25675 tablet_bootstrap.cc:654] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.218266 25675 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:33.218348 25675 ts_tablet_manager.cc:1367] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:33.218730 25675 raft_consensus.cc:348] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.218822 25675 raft_consensus.cc:374] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.218851 25675 raft_consensus.cc:729] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.218993 25675 consensus_queue.cc:260] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.219354 25675 ts_tablet_manager.cc:1398] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:33.229063 25642 raft_consensus.cc:482] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.229193 25642 raft_consensus.cc:504] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.230324 25642 leader_election.cc:288] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:33.234256 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e30faedce3742e0ba83de21059613d1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:33.234454 24857 raft_consensus.cc:2454] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:33.234622 24988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e30faedce3742e0ba83de21059613d1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:33.234766 24988 raft_consensus.cc:2454] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:33.234804 24530 leader_election.cc:302] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:33.235082 25642 raft_consensus.cc:2781] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.235162 25642 raft_consensus.cc:482] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.235206 25642 raft_consensus.cc:3037] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.236039 25642 raft_consensus.cc:504] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.236411 25642 leader_election.cc:288] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:33.236794 24988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e30faedce3742e0ba83de21059613d1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:33.236776 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e30faedce3742e0ba83de21059613d1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:33.236898 24988 raft_consensus.cc:3037] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.236899 24857 raft_consensus.cc:3037] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.237911 24988 raft_consensus.cc:2454] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:33.237911 24857 raft_consensus.cc:2454] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:33.238210 24530 leader_election.cc:302] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:33.238407 25642 raft_consensus.cc:2781] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.238502 25642 raft_consensus.cc:686] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:33.238795 25642 consensus_queue.cc:237] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:33.241281 24451 catalog_manager.cc:5484] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:33.376305 25642 raft_consensus.cc:482] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.376462 25642 raft_consensus.cc:504] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.376878 25642 leader_election.cc:288] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:33.377365 24988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a37dfc7a4324e2b8c43acb51d07a277" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:33.377415 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a37dfc7a4324e2b8c43acb51d07a277" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:33.377521 24988 raft_consensus.cc:2454] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:33.377537 24857 raft_consensus.cc:2454] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:33.377808 24530 leader_election.cc:302] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:33.377991 25642 raft_consensus.cc:2781] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.378048 25642 raft_consensus.cc:482] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.378075 25642 raft_consensus.cc:3037] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.378890 25642 raft_consensus.cc:504] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.379220 25642 leader_election.cc:288] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:33.379642 24988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a37dfc7a4324e2b8c43acb51d07a277" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:33.379642 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a37dfc7a4324e2b8c43acb51d07a277" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:33.379760 24988 raft_consensus.cc:3037] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.379760 24857 raft_consensus.cc:3037] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.380652 24857 raft_consensus.cc:2454] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:33.380649 24988 raft_consensus.cc:2454] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:33.380939 24530 leader_election.cc:302] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:33.381089 25642 raft_consensus.cc:2781] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.381177 25642 raft_consensus.cc:686] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:33.381316 25642 consensus_queue.cc:237] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.382877 24451 catalog_manager.cc:5484] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
W20240501 19:56:33.506578 25691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:33.506817 25691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:33.509900 25691 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:56:33.581461 24596 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.581954 25644 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.583753 24857 raft_consensus.cc:3037] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.584944 24857 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.585211 24988 raft_consensus.cc:1261] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.585667 25644 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.585980 25642 consensus_queue.cc:1035] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.586977 24857 raft_consensus.cc:1261] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.590804 25647 consensus_queue.cc:1035] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.598582 24856 raft_consensus.cc:1261] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.598839 24989 raft_consensus.cc:1261] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.598968 25642 consensus_queue.cc:1035] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.601596 25642 consensus_queue.cc:1035] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.612216 25610 mvcc.cc:205] Tried to move back new op lower bound from 7022974540117581824 to 7022974539291193344. Current Snapshot: MvccSnapshot[applied={T|T < 7022974540117581824}]
INSERT report
rows total: 2000
time total: 101.614 ms
time per row: 0.0508069 ms
I20240501 19:56:33.651021 24445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:33.656458 24968 tablet_service.cc:1467] Processing CreateTablet for tablet ceed9cadcaa947b995ebc07e0e05fae9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.656524 24707 tablet_service.cc:1467] Processing CreateTablet for tablet ceed9cadcaa947b995ebc07e0e05fae9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.656821 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceed9cadcaa947b995ebc07e0e05fae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.656821 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceed9cadcaa947b995ebc07e0e05fae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.657142 24576 tablet_service.cc:1467] Processing CreateTablet for tablet ceed9cadcaa947b995ebc07e0e05fae9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.657429 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceed9cadcaa947b995ebc07e0e05fae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.658088 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 382da83035804db1bc8743d1916bc376 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.658308 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382da83035804db1bc8743d1916bc376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.659082 25672 tablet_bootstrap.cc:492] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:33.659626 25671 tablet_bootstrap.cc:492] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:33.659914 25672 tablet_bootstrap.cc:654] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.660274 24969 tablet_service.cc:1467] Processing CreateTablet for tablet afc342897aa343fc930be1d27a2bb641 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.660506 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc342897aa343fc930be1d27a2bb641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.660532 25671 tablet_bootstrap.cc:654] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.660542 25673 tablet_bootstrap.cc:492] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:33.660933 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 382da83035804db1bc8743d1916bc376 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.661159 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382da83035804db1bc8743d1916bc376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.661448 25673 tablet_bootstrap.cc:654] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.661489 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 382da83035804db1bc8743d1916bc376 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.661722 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382da83035804db1bc8743d1916bc376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.662392 24836 tablet_service.cc:1467] Processing CreateTablet for tablet afc342897aa343fc930be1d27a2bb641 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.662669 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc342897aa343fc930be1d27a2bb641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.663884 24707 tablet_service.cc:1467] Processing CreateTablet for tablet afc342897aa343fc930be1d27a2bb641 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.664098 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc342897aa343fc930be1d27a2bb641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.665328 25675 tablet_bootstrap.cc:492] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:33.665777 25672 tablet_bootstrap.cc:492] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:33.665869 25672 ts_tablet_manager.cc:1367] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:33.666102 25675 tablet_bootstrap.cc:654] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.666365 25672 raft_consensus.cc:348] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.666482 25672 raft_consensus.cc:374] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.666533 25672 raft_consensus.cc:729] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.666646 25673 tablet_bootstrap.cc:492] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:33.666667 25672 consensus_queue.cc:260] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.666743 25673 ts_tablet_manager.cc:1367] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:33.666978 25672 ts_tablet_manager.cc:1398] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.667234 25673 raft_consensus.cc:348] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.667335 25673 raft_consensus.cc:374] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.667369 25673 raft_consensus.cc:729] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.667447 25675 tablet_bootstrap.cc:492] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:33.667516 25675 ts_tablet_manager.cc:1367] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:33.667495 25673 consensus_queue.cc:260] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.667541 25672 tablet_bootstrap.cc:492] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:33.667762 25673 ts_tablet_manager.cc:1398] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.667903 25673 tablet_bootstrap.cc:492] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:33.667929 25675 raft_consensus.cc:348] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.668009 25675 raft_consensus.cc:374] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.668038 25675 raft_consensus.cc:729] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.668140 25675 consensus_queue.cc:260] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.668401 25675 ts_tablet_manager.cc:1398] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:33.668788 25672 tablet_bootstrap.cc:654] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.670006 25673 tablet_bootstrap.cc:654] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.670322 25671 tablet_bootstrap.cc:492] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:33.670418 25671 ts_tablet_manager.cc:1367] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240501 19:56:33.670902 25671 raft_consensus.cc:348] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.670998 25671 raft_consensus.cc:374] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.671036 25671 raft_consensus.cc:729] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.671097 25672 tablet_bootstrap.cc:492] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:33.671183 25672 ts_tablet_manager.cc:1367] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:33.671208 25671 consensus_queue.cc:260] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.671519 25671 ts_tablet_manager.cc:1398] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.671661 25671 tablet_bootstrap.cc:492] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:33.671623 25672 raft_consensus.cc:348] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.671720 25672 raft_consensus.cc:374] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.671756 25672 raft_consensus.cc:729] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.671881 25672 consensus_queue.cc:260] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.672152 25672 ts_tablet_manager.cc:1398] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:33.672260 25673 tablet_bootstrap.cc:492] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:33.672325 25673 ts_tablet_manager.cc:1367] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:33.672334 25672 tablet_bootstrap.cc:492] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:33.672551 25671 tablet_bootstrap.cc:654] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.672772 25673 raft_consensus.cc:348] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.672861 25673 raft_consensus.cc:374] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.672899 25673 raft_consensus.cc:729] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.673013 25673 consensus_queue.cc:260] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.673275 25672 tablet_bootstrap.cc:654] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.673295 25673 ts_tablet_manager.cc:1398] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.673470 25673 tablet_bootstrap.cc:492] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:33.674329 25673 tablet_bootstrap.cc:654] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.674352 25672 tablet_bootstrap.cc:492] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:33.674422 25672 ts_tablet_manager.cc:1367] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:33.674829 25672 raft_consensus.cc:348] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.674964 25672 raft_consensus.cc:374] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.674993 25672 raft_consensus.cc:729] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.675189 25672 consensus_queue.cc:260] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.675464 25672 ts_tablet_manager.cc:1398] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.676427 25671 tablet_bootstrap.cc:492] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:33.676646 25671 ts_tablet_manager.cc:1367] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:33.676659 25673 tablet_bootstrap.cc:492] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:33.676731 25673 ts_tablet_manager.cc:1367] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:33.677121 25673 raft_consensus.cc:348] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.677134 25671 raft_consensus.cc:348] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.677227 25671 raft_consensus.cc:374] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.677227 25673 raft_consensus.cc:374] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.677259 25671 raft_consensus.cc:729] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.677259 25673 raft_consensus.cc:729] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.677376 25673 consensus_queue.cc:260] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.677407 25671 consensus_queue.cc:260] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.677615 25673 ts_tablet_manager.cc:1398] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:33.677908 25671 ts_tablet_manager.cc:1398] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:33.728585 25669 raft_consensus.cc:482] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.728729 25669 raft_consensus.cc:504] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.729171 25669 leader_election.cc:288] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:33.729614 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc342897aa343fc930be1d27a2bb641" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:33.729619 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc342897aa343fc930be1d27a2bb641" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:33.729755 24989 raft_consensus.cc:2454] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:33.729764 24727 raft_consensus.cc:2454] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:33.730082 24792 leader_election.cc:302] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:33.730232 25669 raft_consensus.cc:2781] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.730288 25669 raft_consensus.cc:482] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.730355 25669 raft_consensus.cc:3037] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.731177 25669 raft_consensus.cc:504] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.731528 25669 leader_election.cc:288] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:33.731802 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc342897aa343fc930be1d27a2bb641" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:33.731840 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc342897aa343fc930be1d27a2bb641" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:33.731914 24727 raft_consensus.cc:3037] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.731940 24989 raft_consensus.cc:3037] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.732748 24989 raft_consensus.cc:2454] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:33.732780 24727 raft_consensus.cc:2454] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:33.732991 24792 leader_election.cc:302] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:33.733153 25669 raft_consensus.cc:2781] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.733238 25669 raft_consensus.cc:686] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:33.733382 25669 consensus_queue.cc:237] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:33.734988 24445 catalog_manager.cc:5484] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:33.786680 25651 raft_consensus.cc:482] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.786816 25651 raft_consensus.cc:504] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.787896 25651 leader_election.cc:288] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:33.791837 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382da83035804db1bc8743d1916bc376" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:33.792008 24595 raft_consensus.cc:2454] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:33.792021 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382da83035804db1bc8743d1916bc376" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:33.792168 24727 raft_consensus.cc:2454] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:33.792450 24922 leader_election.cc:302] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:33.792718 25651 raft_consensus.cc:2781] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.792790 25651 raft_consensus.cc:482] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.792840 25651 raft_consensus.cc:3037] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.793800 25651 raft_consensus.cc:504] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.794111 25651 leader_election.cc:288] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:33.794437 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382da83035804db1bc8743d1916bc376" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:33.794481 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382da83035804db1bc8743d1916bc376" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:33.794541 24595 raft_consensus.cc:3037] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.794605 24727 raft_consensus.cc:3037] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.795374 24595 raft_consensus.cc:2454] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:33.795409 24727 raft_consensus.cc:2454] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:33.795683 24922 leader_election.cc:302] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:33.795833 25651 raft_consensus.cc:2781] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.795928 25651 raft_consensus.cc:686] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:33.796208 25651 consensus_queue.cc:237] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:33.798228 24445 catalog_manager.cc:5484] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:33.856129 25651 raft_consensus.cc:482] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.856345 25651 raft_consensus.cc:504] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.856778 25651 leader_election.cc:288] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:33.857251 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceed9cadcaa947b995ebc07e0e05fae9" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:33.857306 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceed9cadcaa947b995ebc07e0e05fae9" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:33.857405 24727 raft_consensus.cc:2454] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:33.857484 24595 raft_consensus.cc:2454] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:33.857757 24922 leader_election.cc:302] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:33.858070 25651 raft_consensus.cc:2781] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.858126 25651 raft_consensus.cc:482] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.858155 25651 raft_consensus.cc:3037] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.859123 25651 raft_consensus.cc:504] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.859459 25651 leader_election.cc:288] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:33.859764 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceed9cadcaa947b995ebc07e0e05fae9" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:33.859862 24727 raft_consensus.cc:3037] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.859879 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceed9cadcaa947b995ebc07e0e05fae9" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:33.859993 24595 raft_consensus.cc:3037] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.860615 24727 raft_consensus.cc:2454] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:33.860761 24595 raft_consensus.cc:2454] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:33.860864 24922 leader_election.cc:302] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:33.861095 25651 raft_consensus.cc:2781] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.861172 25651 raft_consensus.cc:686] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:33.861300 25651 consensus_queue.cc:237] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:33.862797 24445 catalog_manager.cc:5484] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
W20240501 19:56:34.013031 25743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:34.013401 25743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:34.016422 25743 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:56:34.104426 24989 raft_consensus.cc:1261] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.104990 25669 consensus_queue.cc:1035] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.105810 24595 raft_consensus.cc:1261] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.106303 25651 consensus_queue.cc:1035] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.108364 24594 raft_consensus.cc:1261] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.108731 24727 raft_consensus.cc:1261] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.109036 24726 raft_consensus.cc:1261] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.109987 24726 raft_consensus.cc:1261] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.110525 25594 consensus_queue.cc:1035] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.110695 25651 consensus_queue.cc:1035] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.110823 25594 consensus_queue.cc:1035] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.111747 25602 consensus_queue.cc:1035] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.165 ms
time per row: 0.0510826 ms
I20240501 19:56:34.167104 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:34.172528 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 39e7bc7629944b3da3fe8fb1ecae32b1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.172528 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 39e7bc7629944b3da3fe8fb1ecae32b1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.172827 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e7bc7629944b3da3fe8fb1ecae32b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.172824 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e7bc7629944b3da3fe8fb1ecae32b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.172813 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 39e7bc7629944b3da3fe8fb1ecae32b1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.173102 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e7bc7629944b3da3fe8fb1ecae32b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.174877 25791 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:34.174934 25671 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:34.175484 25673 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:34.175799 25791 tablet_bootstrap.cc:654] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.175802 25671 tablet_bootstrap.cc:654] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.176400 25673 tablet_bootstrap.cc:654] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.177089 25671 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:34.177186 25671 ts_tablet_manager.cc:1367] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:34.177656 25671 raft_consensus.cc:348] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.177652 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2603a318c34256a9fa28979ae41fec (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.177652 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 66215de525314ea180c164667f4e9f3f (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.177768 25671 raft_consensus.cc:374] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.177803 25671 raft_consensus.cc:729] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.177908 25671 consensus_queue.cc:260] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.177954 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66215de525314ea180c164667f4e9f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.178058 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2603a318c34256a9fa28979ae41fec (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.178169 25671 ts_tablet_manager.cc:1398] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.178292 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2603a318c34256a9fa28979ae41fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.178695 25791 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:34.178781 25791 ts_tablet_manager.cc:1367] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:34.179425 25791 raft_consensus.cc:348] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.179543 25791 raft_consensus.cc:374] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.179582 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 66215de525314ea180c164667f4e9f3f (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.179844 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66215de525314ea180c164667f4e9f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.180568 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2603a318c34256a9fa28979ae41fec (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.180801 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2603a318c34256a9fa28979ae41fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.179579 25791 raft_consensus.cc:729] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.181851 25791 consensus_queue.cc:260] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.182366 25673 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:34.182454 25673 ts_tablet_manager.cc:1367] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:56:34.182634 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 66215de525314ea180c164667f4e9f3f (DEFAULT_TABLE table=rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.182713 25791 ts_tablet_manager.cc:1398] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:34.182858 25791 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:34.182869 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66215de525314ea180c164667f4e9f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.182955 25671 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:34.182870 25673 raft_consensus.cc:348] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.183089 25673 raft_consensus.cc:374] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.183121 25673 raft_consensus.cc:729] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.183243 25673 consensus_queue.cc:260] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.183475 25673 ts_tablet_manager.cc:1398] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.183770 25791 tablet_bootstrap.cc:654] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.183770 25671 tablet_bootstrap.cc:654] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.184845 25673 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:34.185662 25673 tablet_bootstrap.cc:654] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.184846 25671 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:34.186640 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2603a318c34256a9fa28979ae41fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.186648 25671 ts_tablet_manager.cc:1367] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:34.186739 25791 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:34.186820 25791 ts_tablet_manager.cc:1367] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:34.187127 25671 raft_consensus.cc:348] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.187273 25671 raft_consensus.cc:374] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.187309 25671 raft_consensus.cc:729] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.187258 25791 raft_consensus.cc:348] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.187352 25791 raft_consensus.cc:374] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.187382 25791 raft_consensus.cc:729] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.187429 25671 consensus_queue.cc:260] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.187505 25791 consensus_queue.cc:260] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.187696 25671 ts_tablet_manager.cc:1398] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.187829 25791 ts_tablet_manager.cc:1398] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.187988 25791 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:34.188334 25673 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:34.188442 25673 ts_tablet_manager.cc:1367] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:34.188793 25795 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:34.188824 25791 tablet_bootstrap.cc:654] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.188843 25673 raft_consensus.cc:348] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.188962 25673 raft_consensus.cc:374] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.189011 25673 raft_consensus.cc:729] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.189129 25673 consensus_queue.cc:260] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.189383 25673 ts_tablet_manager.cc:1398] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.189851 25795 tablet_bootstrap.cc:654] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.190428 25791 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:34.190502 25791 ts_tablet_manager.cc:1367] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:34.190888 25791 raft_consensus.cc:348] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.190994 25791 raft_consensus.cc:374] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.191054 25791 raft_consensus.cc:729] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.191177 25791 consensus_queue.cc:260] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.191421 25791 ts_tablet_manager.cc:1398] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:34.191988 25795 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:34.192090 25795 ts_tablet_manager.cc:1367] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:34.192484 25795 raft_consensus.cc:348] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.192561 25795 raft_consensus.cc:374] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.192581 25795 raft_consensus.cc:729] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.192811 25795 consensus_queue.cc:260] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.192951 25774 raft_consensus.cc:482] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:34.193027 25774 raft_consensus.cc:504] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.193344 25774 leader_election.cc:288] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:34.194051 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66215de525314ea180c164667f4e9f3f" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:34.194177 24726 raft_consensus.cc:2454] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:34.194456 24922 leader_election.cc:302] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:34.194592 25774 raft_consensus.cc:2781] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:34.194644 25774 raft_consensus.cc:482] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:34.194677 25774 raft_consensus.cc:3037] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.195417 25774 raft_consensus.cc:504] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.195763 25774 leader_election.cc:288] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:34.196045 25795 ts_tablet_manager.cc:1398] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:34.196110 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66215de525314ea180c164667f4e9f3f" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:34.196199 24726 raft_consensus.cc:3037] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.196218 25795 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:34.197033 24726 raft_consensus.cc:2454] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:34.197145 25795 tablet_bootstrap.cc:654] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.197304 24922 leader_election.cc:302] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:34.197471 25774 raft_consensus.cc:2781] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.197553 25774 raft_consensus.cc:686] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:34.197717 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66215de525314ea180c164667f4e9f3f" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:34.197695 25774 consensus_queue.cc:237] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:34.197815 24857 raft_consensus.cc:2454] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:34.197993 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66215de525314ea180c164667f4e9f3f" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:34.198079 24856 raft_consensus.cc:3037] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.198690 25795 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:34.198762 25795 ts_tablet_manager.cc:1367] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:34.199079 24856 raft_consensus.cc:2454] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:34.199231 25795 raft_consensus.cc:348] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.199326 25795 raft_consensus.cc:374] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.199390 25795 raft_consensus.cc:729] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.199525 25795 consensus_queue.cc:260] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.199580 24449 catalog_manager.cc:5484] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:34.199817 25795 ts_tablet_manager.cc:1398] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.248137 25722 raft_consensus.cc:482] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:34.248273 25722 raft_consensus.cc:504] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.248826 25722 leader_election.cc:288] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:34.249498 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7bc7629944b3da3fe8fb1ecae32b1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:34.249673 24856 raft_consensus.cc:2454] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:34.250134 24530 leader_election.cc:302] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:34.250329 25722 raft_consensus.cc:2781] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:34.250411 25722 raft_consensus.cc:482] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:34.250458 25722 raft_consensus.cc:3037] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.251466 25722 raft_consensus.cc:504] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.251816 25722 leader_election.cc:288] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:34.252218 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7bc7629944b3da3fe8fb1ecae32b1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:34.252341 24856 raft_consensus.cc:3037] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.252990 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7bc7629944b3da3fe8fb1ecae32b1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:34.253041 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7bc7629944b3da3fe8fb1ecae32b1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:34.253110 24726 raft_consensus.cc:2454] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:34.253297 24856 raft_consensus.cc:2454] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:34.253556 24530 leader_election.cc:302] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters: 9081292597be4a72a42b216fddcd7971
I20240501 19:56:34.253721 25722 raft_consensus.cc:2781] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.253898 25722 raft_consensus.cc:686] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:34.254032 25722 consensus_queue.cc:237] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.255641 24449 catalog_manager.cc:5484] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:34.468252 25669 raft_consensus.cc:482] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:34.468403 25669 raft_consensus.cc:504] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.468839 25669 leader_election.cc:288] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:34.469281 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2603a318c34256a9fa28979ae41fec" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:34.469426 24595 raft_consensus.cc:2454] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:34.469715 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2603a318c34256a9fa28979ae41fec" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:34.469765 24791 leader_election.cc:302] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:34.469866 24989 raft_consensus.cc:2454] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:34.469946 25669 raft_consensus.cc:2781] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:34.470008 25669 raft_consensus.cc:482] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:34.470065 25669 raft_consensus.cc:3037] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.471021 25669 raft_consensus.cc:504] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.471405 25669 leader_election.cc:288] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:34.471778 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2603a318c34256a9fa28979ae41fec" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:34.471778 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2603a318c34256a9fa28979ae41fec" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:34.471899 24595 raft_consensus.cc:3037] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.471899 24989 raft_consensus.cc:3037] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.472807 24595 raft_consensus.cc:2454] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:34.472807 24989 raft_consensus.cc:2454] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:34.473083 24792 leader_election.cc:302] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:34.473230 25669 raft_consensus.cc:2781] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.473307 25669 raft_consensus.cc:686] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:34.473441 25669 consensus_queue.cc:237] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.475304 24449 catalog_manager.cc:5484] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:34.632908 25773 consensus_queue.cc:1035] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.636097 25774 consensus_queue.cc:1035] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:34.654484 25807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:34.654729 25807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:34.657763 25807 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:56:34.710985 24725 raft_consensus.cc:3037] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.712604 25722 consensus_queue.cc:1035] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.727985 25804 consensus_queue.cc:1035] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.749565 24989 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.750048 25669 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.758728 24595 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.759158 25669 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 131.688 ms
time per row: 0.065844 ms
I20240501 19:56:34.830442 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:34.835743 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1f471ee79f41289c961066603539cb (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.835793 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1f471ee79f41289c961066603539cb (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.835827 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1f471ee79f41289c961066603539cb (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.836077 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1f471ee79f41289c961066603539cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.836077 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1f471ee79f41289c961066603539cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.836081 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1f471ee79f41289c961066603539cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.836930 24836 tablet_service.cc:1467] Processing CreateTablet for tablet d784f58f5a314629a1f8449f11962fb6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.836964 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 01905e6ce63e4a67bc1ac51f84a05b74 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.836984 24969 tablet_service.cc:1467] Processing CreateTablet for tablet d784f58f5a314629a1f8449f11962fb6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.837195 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01905e6ce63e4a67bc1ac51f84a05b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.837347 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d784f58f5a314629a1f8449f11962fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.837981 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d784f58f5a314629a1f8449f11962fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.839315 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 01905e6ce63e4a67bc1ac51f84a05b74 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.839540 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01905e6ce63e4a67bc1ac51f84a05b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.839932 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 01905e6ce63e4a67bc1ac51f84a05b74 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.840174 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01905e6ce63e4a67bc1ac51f84a05b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.840373 25865 tablet_bootstrap.cc:492] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:34.841315 25865 tablet_bootstrap.cc:654] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.841548 24705 tablet_service.cc:1467] Processing CreateTablet for tablet d784f58f5a314629a1f8449f11962fb6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.841763 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d784f58f5a314629a1f8449f11962fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.841974 25866 tablet_bootstrap.cc:492] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:34.842342 25864 tablet_bootstrap.cc:492] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:34.842800 25866 tablet_bootstrap.cc:654] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.843243 25864 tablet_bootstrap.cc:654] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.843261 25863 tablet_bootstrap.cc:492] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:34.844157 25863 tablet_bootstrap.cc:654] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.844295 25866 tablet_bootstrap.cc:492] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:34.844386 25866 ts_tablet_manager.cc:1367] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:34.844516 25864 tablet_bootstrap.cc:492] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:34.844601 25864 ts_tablet_manager.cc:1367] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:34.844834 25866 raft_consensus.cc:348] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.844954 25866 raft_consensus.cc:374] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.844990 25866 raft_consensus.cc:729] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.845065 25864 raft_consensus.cc:348] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.845110 25866 consensus_queue.cc:260] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.845162 25864 raft_consensus.cc:374] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.845220 25864 raft_consensus.cc:729] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.845257 25865 tablet_bootstrap.cc:492] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:34.845333 25865 ts_tablet_manager.cc:1367] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:34.845351 25864 consensus_queue.cc:260] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.845387 25866 ts_tablet_manager.cc:1398] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:34.845544 25866 tablet_bootstrap.cc:492] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:34.845598 25864 ts_tablet_manager.cc:1398] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.845723 25864 tablet_bootstrap.cc:492] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:34.845765 25865 raft_consensus.cc:348] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.845854 25865 raft_consensus.cc:374] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.845891 25865 raft_consensus.cc:729] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.846024 25865 consensus_queue.cc:260] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.846284 25865 ts_tablet_manager.cc:1398] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:34.846371 25866 tablet_bootstrap.cc:654] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.846439 25864 tablet_bootstrap.cc:654] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.846442 25865 tablet_bootstrap.cc:492] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:34.847412 25865 tablet_bootstrap.cc:654] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.847446 25866 tablet_bootstrap.cc:492] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:34.847517 25866 ts_tablet_manager.cc:1367] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:34.847934 25866 raft_consensus.cc:348] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.848016 25866 raft_consensus.cc:374] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.848047 25866 raft_consensus.cc:729] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.848162 25866 consensus_queue.cc:260] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.848419 25866 ts_tablet_manager.cc:1398] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:34.850884 25863 tablet_bootstrap.cc:492] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:34.850934 25865 tablet_bootstrap.cc:492] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:34.850970 25863 ts_tablet_manager.cc:1367] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:34.851084 25865 ts_tablet_manager.cc:1367] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:34.851294 25864 tablet_bootstrap.cc:492] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:34.851374 25864 ts_tablet_manager.cc:1367] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:34.851393 25863 raft_consensus.cc:348] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.851497 25863 raft_consensus.cc:374] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.851516 25865 raft_consensus.cc:348] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.851559 25863 raft_consensus.cc:729] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.851613 25865 raft_consensus.cc:374] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.851681 25865 raft_consensus.cc:729] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.851699 25863 consensus_queue.cc:260] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.851802 25865 consensus_queue.cc:260] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:34.851989 25863 ts_tablet_manager.cc:1398] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.852118 25865 ts_tablet_manager.cc:1398] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.852232 25863 tablet_bootstrap.cc:492] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:34.851819 25864 raft_consensus.cc:348] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.852494 25864 raft_consensus.cc:374] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.852586 25864 raft_consensus.cc:729] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.853286 25864 consensus_queue.cc:260] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.853633 25864 ts_tablet_manager.cc:1398] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:34.853832 25863 tablet_bootstrap.cc:654] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.855501 25863 tablet_bootstrap.cc:492] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:34.855587 25863 ts_tablet_manager.cc:1367] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:34.855983 25863 raft_consensus.cc:348] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.856068 25863 raft_consensus.cc:374] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.856304 25863 raft_consensus.cc:729] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.856480 25863 consensus_queue.cc:260] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.856747 25863 ts_tablet_manager.cc:1398] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.856933 25863 tablet_bootstrap.cc:492] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:34.857605 25863 tablet_bootstrap.cc:654] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.858484 25863 tablet_bootstrap.cc:492] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:34.858546 25863 ts_tablet_manager.cc:1367] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:34.858909 25863 raft_consensus.cc:348] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.858994 25863 raft_consensus.cc:374] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.859014 25863 raft_consensus.cc:729] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.859165 25863 consensus_queue.cc:260] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.859447 25863 ts_tablet_manager.cc:1398] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:34.967787 25773 raft_consensus.cc:482] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:34.967911 25773 raft_consensus.cc:504] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.968261 25773 leader_election.cc:288] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:34.968634 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d784f58f5a314629a1f8449f11962fb6" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:34.968672 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d784f58f5a314629a1f8449f11962fb6" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:34.968806 24725 raft_consensus.cc:2454] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:34.968838 24856 raft_consensus.cc:2454] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:34.969128 24922 leader_election.cc:302] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:34.969276 25773 raft_consensus.cc:2781] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:34.969342 25773 raft_consensus.cc:482] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:34.969383 25773 raft_consensus.cc:3037] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.970201 25773 raft_consensus.cc:504] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.970537 25773 leader_election.cc:288] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:34.970927 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d784f58f5a314629a1f8449f11962fb6" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:34.970932 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d784f58f5a314629a1f8449f11962fb6" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:34.971021 24856 raft_consensus.cc:3037] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.971032 24725 raft_consensus.cc:3037] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.971777 24725 raft_consensus.cc:2454] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:34.972029 24922 leader_election.cc:302] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:34.972082 24856 raft_consensus.cc:2454] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:34.972205 25773 raft_consensus.cc:2781] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.972281 25773 raft_consensus.cc:686] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:34.972450 25773 consensus_queue.cc:237] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:34.974058 24449 catalog_manager.cc:5484] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:35.099140 25851 raft_consensus.cc:482] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.099265 25851 raft_consensus.cc:504] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.099717 25851 leader_election.cc:288] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:35.100106 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1f471ee79f41289c961066603539cb" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:35.100104 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1f471ee79f41289c961066603539cb" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:35.100235 24856 raft_consensus.cc:2454] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:35.100243 24725 raft_consensus.cc:2454] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:35.100546 24530 leader_election.cc:302] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:35.100700 25851 raft_consensus.cc:2781] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.100770 25851 raft_consensus.cc:482] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.100796 25851 raft_consensus.cc:3037] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.101542 25851 raft_consensus.cc:504] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.101842 25851 leader_election.cc:288] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:35.102134 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1f471ee79f41289c961066603539cb" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:35.102195 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1f471ee79f41289c961066603539cb" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:35.102246 24725 raft_consensus.cc:3037] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.102303 24856 raft_consensus.cc:3037] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.102991 24725 raft_consensus.cc:2454] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:35.102998 24856 raft_consensus.cc:2454] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:35.103299 24529 leader_election.cc:302] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:35.103440 25851 raft_consensus.cc:2781] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.103523 25851 raft_consensus.cc:686] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:35.103633 25851 consensus_queue.cc:237] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.105204 24449 catalog_manager.cc:5484] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:35.158622 25773 raft_consensus.cc:482] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.158752 25773 raft_consensus.cc:504] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.159161 25773 leader_election.cc:288] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:35.159623 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01905e6ce63e4a67bc1ac51f84a05b74" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:35.159667 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01905e6ce63e4a67bc1ac51f84a05b74" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:35.159785 24725 raft_consensus.cc:2454] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:35.159806 24595 raft_consensus.cc:2454] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:35.160072 24922 leader_election.cc:302] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:35.160228 25773 raft_consensus.cc:2781] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.160284 25773 raft_consensus.cc:482] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.160333 25773 raft_consensus.cc:3037] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.161167 25773 raft_consensus.cc:504] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.161463 25773 leader_election.cc:288] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:35.161800 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01905e6ce63e4a67bc1ac51f84a05b74" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:35.161877 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01905e6ce63e4a67bc1ac51f84a05b74" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:35.161918 24725 raft_consensus.cc:3037] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.161989 24595 raft_consensus.cc:3037] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.162843 24595 raft_consensus.cc:2454] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:35.162843 24725 raft_consensus.cc:2454] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:35.163141 24922 leader_election.cc:302] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:35.163280 25773 raft_consensus.cc:2781] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.163364 25773 raft_consensus.cc:686] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:35.163508 25773 consensus_queue.cc:237] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.165153 24449 catalog_manager.cc:5484] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
W20240501 19:56:35.315270 25879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:35.315598 25879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:35.318861 25879 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:56:35.408912 24595 raft_consensus.cc:1261] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.409473 25773 consensus_queue.cc:1035] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.410959 24725 raft_consensus.cc:1261] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.411298 24856 raft_consensus.cc:1261] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.411646 25910 consensus_queue.cc:1035] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.413224 25910 consensus_queue.cc:1035] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.416618 24857 raft_consensus.cc:1261] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.416999 24726 raft_consensus.cc:1261] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.417577 25774 consensus_queue.cc:1035] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.417899 25774 consensus_queue.cc:1035] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.425311 24725 raft_consensus.cc:1261] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.425792 25773 consensus_queue.cc:1035] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.946 ms
time per row: 0.0599728 ms
I20240501 19:56:35.484299 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:35.489689 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.489840 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.490044 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.490125 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.490483 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.490734 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.491641 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8f81836b9944aa964a8236912c0183 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.491853 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8f81836b9944aa964a8236912c0183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.492491 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8f81836b9944aa964a8236912c0183 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.492710 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8f81836b9944aa964a8236912c0183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.492872 25945 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:35.492986 24574 tablet_service.cc:1467] Processing CreateTablet for tablet cb0bb0d3396a436c878f56d157f30eda (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.493176 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0bb0d3396a436c878f56d157f30eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.493867 25945 tablet_bootstrap.cc:654] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.494320 24836 tablet_service.cc:1467] Processing CreateTablet for tablet cb0bb0d3396a436c878f56d157f30eda (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.494544 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0bb0d3396a436c878f56d157f30eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.494628 25943 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:35.495477 25943 tablet_bootstrap.cc:654] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.495431 24705 tablet_service.cc:1467] Processing CreateTablet for tablet cb0bb0d3396a436c878f56d157f30eda (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.495697 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0bb0d3396a436c878f56d157f30eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.496491 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8f81836b9944aa964a8236912c0183 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.496713 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8f81836b9944aa964a8236912c0183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.497035 25943 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:35.497112 25943 ts_tablet_manager.cc:1367] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:35.497552 25943 raft_consensus.cc:348] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.497681 25943 raft_consensus.cc:374] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.497725 25943 raft_consensus.cc:729] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.497853 25943 consensus_queue.cc:260] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.498322 25944 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:35.498508 25948 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:35.498740 25943 ts_tablet_manager.cc:1398] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:35.498899 25943 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:35.499297 25948 tablet_bootstrap.cc:654] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.499841 25943 tablet_bootstrap.cc:654] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.500001 25944 tablet_bootstrap.cc:654] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.500646 25945 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:35.500720 25945 ts_tablet_manager.cc:1367] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:35.500919 25943 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:35.500996 25943 ts_tablet_manager.cc:1367] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:35.501024 25945 raft_consensus.cc:348] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.501091 25945 raft_consensus.cc:374] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.501111 25945 raft_consensus.cc:729] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.501219 25945 consensus_queue.cc:260] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.501379 25943 raft_consensus.cc:348] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.501462 25945 ts_tablet_manager.cc:1398] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.501484 25943 raft_consensus.cc:374] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.501521 25943 raft_consensus.cc:729] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.501610 25945 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:35.501639 25943 consensus_queue.cc:260] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.501950 25943 ts_tablet_manager.cc:1398] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.502133 25948 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:35.502202 25948 ts_tablet_manager.cc:1367] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:35.502221 25944 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:35.502275 25944 ts_tablet_manager.cc:1367] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:35.502456 25945 tablet_bootstrap.cc:654] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.502611 25944 raft_consensus.cc:348] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.502683 25944 raft_consensus.cc:374] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.502703 25944 raft_consensus.cc:729] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.502707 25948 raft_consensus.cc:348] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.502808 25948 raft_consensus.cc:374] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.502801 25944 consensus_queue.cc:260] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.502839 25948 raft_consensus.cc:729] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.503063 25944 ts_tablet_manager.cc:1398] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.503208 25944 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:35.502965 25948 consensus_queue.cc:260] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.503468 25948 ts_tablet_manager.cc:1398] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:35.504570 25944 tablet_bootstrap.cc:654] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.504985 25945 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:35.505079 25945 ts_tablet_manager.cc:1367] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:35.505509 25945 raft_consensus.cc:348] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.505622 25945 raft_consensus.cc:374] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.505661 25945 raft_consensus.cc:729] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.505759 25945 consensus_queue.cc:260] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.505950 25945 ts_tablet_manager.cc:1398] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.506081 25945 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:35.506863 25945 tablet_bootstrap.cc:654] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.507259 25944 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:35.507344 25944 ts_tablet_manager.cc:1367] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:35.507794 25944 raft_consensus.cc:348] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.507894 25944 raft_consensus.cc:374] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.507928 25944 raft_consensus.cc:729] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.508071 25945 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:35.508037 25944 consensus_queue.cc:260] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.508141 25945 ts_tablet_manager.cc:1367] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:35.508296 25944 ts_tablet_manager.cc:1398] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:35.508464 25944 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:35.508836 25945 raft_consensus.cc:348] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.508991 25945 raft_consensus.cc:374] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.509069 25945 raft_consensus.cc:729] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.509255 25945 consensus_queue.cc:260] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.509497 25944 tablet_bootstrap.cc:654] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.509577 25945 ts_tablet_manager.cc:1398] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.510831 25944 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:35.510922 25944 ts_tablet_manager.cc:1367] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:35.511303 25944 raft_consensus.cc:348] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.511409 25944 raft_consensus.cc:374] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.511451 25944 raft_consensus.cc:729] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.511561 25944 consensus_queue.cc:260] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.511829 25944 ts_tablet_manager.cc:1398] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:35.517814 25929 raft_consensus.cc:482] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.517906 25929 raft_consensus.cc:504] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.518210 25929 leader_election.cc:288] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:35.518553 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a4aa3f94bf4a3b8d84cae1e1765ac9" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:35.518604 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a4aa3f94bf4a3b8d84cae1e1765ac9" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:35.518688 24857 raft_consensus.cc:2454] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:35.518719 24989 raft_consensus.cc:2454] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:35.518956 24530 leader_election.cc:302] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:35.519122 25929 raft_consensus.cc:2781] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.519176 25929 raft_consensus.cc:482] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.519205 25929 raft_consensus.cc:3037] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.520160 25929 raft_consensus.cc:504] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.520470 25929 leader_election.cc:288] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:35.520742 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a4aa3f94bf4a3b8d84cae1e1765ac9" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:35.520802 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a4aa3f94bf4a3b8d84cae1e1765ac9" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:35.520852 24857 raft_consensus.cc:3037] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.520903 24989 raft_consensus.cc:3037] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.521772 24989 raft_consensus.cc:2454] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:35.521772 24857 raft_consensus.cc:2454] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:35.522017 24530 leader_election.cc:302] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:35.522171 25929 raft_consensus.cc:2781] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.522255 25929 raft_consensus.cc:686] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:35.522393 25929 consensus_queue.cc:237] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:35.523968 24450 catalog_manager.cc:5484] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:35.550550 25727 raft_consensus.cc:482] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.550726 25727 raft_consensus.cc:504] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.551209 25727 leader_election.cc:288] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:35.551615 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0bb0d3396a436c878f56d157f30eda" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:35.551649 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0bb0d3396a436c878f56d157f30eda" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:35.551741 24595 raft_consensus.cc:2454] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:35.551807 24857 raft_consensus.cc:2454] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:35.552052 24660 leader_election.cc:302] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:35.552269 25727 raft_consensus.cc:2781] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.552316 25727 raft_consensus.cc:482] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.552342 25727 raft_consensus.cc:3037] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.553171 25727 raft_consensus.cc:504] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.553473 25727 leader_election.cc:288] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:35.553782 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0bb0d3396a436c878f56d157f30eda" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:35.553884 24857 raft_consensus.cc:3037] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.554723 24857 raft_consensus.cc:2454] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:35.554971 24661 leader_election.cc:302] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:35.555128 25727 raft_consensus.cc:2781] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.555430 25727 raft_consensus.cc:686] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:35.555594 25727 consensus_queue.cc:237] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.557261 24450 catalog_manager.cc:5484] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:35.559324 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0bb0d3396a436c878f56d157f30eda" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:35.559474 24595 raft_consensus.cc:3037] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.575887 24595 raft_consensus.cc:2454] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:35.738338 25862 raft_consensus.cc:482] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.738487 25862 raft_consensus.cc:504] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.738968 25862 leader_election.cc:288] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:35.739434 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8f81836b9944aa964a8236912c0183" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:35.739430 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8f81836b9944aa964a8236912c0183" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:35.739614 24989 raft_consensus.cc:2454] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:35.739616 24595 raft_consensus.cc:2454] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:35.739930 24791 leader_election.cc:302] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:35.740120 25862 raft_consensus.cc:2781] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.740187 25862 raft_consensus.cc:482] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.740238 25862 raft_consensus.cc:3037] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.741153 25862 raft_consensus.cc:504] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.741520 25862 leader_election.cc:288] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:35.741850 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8f81836b9944aa964a8236912c0183" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:35.741850 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8f81836b9944aa964a8236912c0183" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:35.741978 24595 raft_consensus.cc:3037] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.741978 24989 raft_consensus.cc:3037] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.742936 24595 raft_consensus.cc:2454] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:35.742936 24989 raft_consensus.cc:2454] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:35.743247 24791 leader_election.cc:302] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:35.743598 25862 raft_consensus.cc:2781] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.743817 25862 raft_consensus.cc:686] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:35.743963 25862 consensus_queue.cc:237] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:35.745576 24450 catalog_manager.cc:5484] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
W20240501 19:56:35.903067 25961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:35.903373 25961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:35.906396 25961 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 19:56:35.959761 25727 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.963204 25957 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.978824 25929 consensus_queue.cc:1035] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.982851 25929 consensus_queue.cc:1035] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.009080 24595 raft_consensus.cc:1261] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.011003 24989 raft_consensus.cc:1261] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.011379 25959 consensus_queue.cc:1035] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.011682 25959 consensus_queue.cc:1035] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.996 ms
time per row: 0.056498 ms
I20240501 19:56:36.067919 24445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:36.073457 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 9405d611c5a04984bb7cfb24f25d768d (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.073457 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 9405d611c5a04984bb7cfb24f25d768d (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.073784 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9405d611c5a04984bb7cfb24f25d768d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.073784 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9405d611c5a04984bb7cfb24f25d768d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.073839 24706 tablet_service.cc:1467] Processing CreateTablet for tablet a1cf6622eed045e7a864b9e397618ecd (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.074072 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cf6622eed045e7a864b9e397618ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.074332 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 9405d611c5a04984bb7cfb24f25d768d (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.074599 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9405d611c5a04984bb7cfb24f25d768d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.075927 26018 tablet_bootstrap.cc:492] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:36.076483 24969 tablet_service.cc:1467] Processing CreateTablet for tablet a1cf6622eed045e7a864b9e397618ecd (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.076508 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 6c35e1be84514add973014e564ad845c (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.076750 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cf6622eed045e7a864b9e397618ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.076836 26018 tablet_bootstrap.cc:654] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.077054 24838 tablet_service.cc:1467] Processing CreateTablet for tablet a1cf6622eed045e7a864b9e397618ecd (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.077327 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cf6622eed045e7a864b9e397618ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.077548 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 6c35e1be84514add973014e564ad845c (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.077575 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 6c35e1be84514add973014e564ad845c (DEFAULT_TABLE table=rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.077764 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c35e1be84514add973014e564ad845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.077769 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c35e1be84514add973014e564ad845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.079260 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c35e1be84514add973014e564ad845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.079605 26019 tablet_bootstrap.cc:492] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:36.079665 26018 tablet_bootstrap.cc:492] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:36.079761 26018 ts_tablet_manager.cc:1367] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:36.080188 26018 raft_consensus.cc:348] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.080281 26018 raft_consensus.cc:374] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.080315 26018 raft_consensus.cc:729] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.080477 26018 consensus_queue.cc:260] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.080581 26019 tablet_bootstrap.cc:654] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.080706 26021 tablet_bootstrap.cc:492] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:36.081379 26018 ts_tablet_manager.cc:1398] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:36.081569 26018 tablet_bootstrap.cc:492] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:36.081570 26021 tablet_bootstrap.cc:654] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.082460 26019 tablet_bootstrap.cc:492] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:36.082571 26019 ts_tablet_manager.cc:1367] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:36.082962 26018 tablet_bootstrap.cc:654] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.083029 26019 raft_consensus.cc:348] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.083220 26019 raft_consensus.cc:374] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.083261 26019 raft_consensus.cc:729] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.083433 26019 consensus_queue.cc:260] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.083712 26019 ts_tablet_manager.cc:1398] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:36.083909 26019 tablet_bootstrap.cc:492] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:36.084339 26021 tablet_bootstrap.cc:492] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:36.084445 26021 ts_tablet_manager.cc:1367] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:36.084640 26018 tablet_bootstrap.cc:492] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:36.084707 26018 ts_tablet_manager.cc:1367] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:36.084810 26019 tablet_bootstrap.cc:654] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.084903 26021 raft_consensus.cc:348] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.084996 26021 raft_consensus.cc:374] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.085027 26021 raft_consensus.cc:729] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.085107 26018 raft_consensus.cc:348] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.085193 26018 raft_consensus.cc:374] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.085224 26018 raft_consensus.cc:729] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.085178 26021 consensus_queue.cc:260] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.085337 26018 consensus_queue.cc:260] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.085476 26021 ts_tablet_manager.cc:1398] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.085606 26018 ts_tablet_manager.cc:1398] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.085772 26018 tablet_bootstrap.cc:492] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:36.087294 26019 tablet_bootstrap.cc:492] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:36.087436 26019 ts_tablet_manager.cc:1367] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:36.087956 26018 tablet_bootstrap.cc:654] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.087924 26019 raft_consensus.cc:348] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.088101 26019 raft_consensus.cc:374] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.088193 26019 raft_consensus.cc:729] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.088397 26019 consensus_queue.cc:260] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.088699 26019 ts_tablet_manager.cc:1398] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.089329 26018 tablet_bootstrap.cc:492] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:36.089407 26018 ts_tablet_manager.cc:1367] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:36.089816 26018 raft_consensus.cc:348] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.089910 26018 raft_consensus.cc:374] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.089953 26018 raft_consensus.cc:729] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.090077 26018 consensus_queue.cc:260] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.090342 26018 ts_tablet_manager.cc:1398] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.092978 26027 tablet_bootstrap.cc:492] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:36.093832 26027 tablet_bootstrap.cc:654] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.095144 26027 tablet_bootstrap.cc:492] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:36.095216 26027 ts_tablet_manager.cc:1367] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:36.095572 26027 raft_consensus.cc:348] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.095664 26027 raft_consensus.cc:374] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.095685 26027 raft_consensus.cc:729] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.095798 26027 consensus_queue.cc:260] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.096086 26027 ts_tablet_manager.cc:1398] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:36.096256 26027 tablet_bootstrap.cc:492] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:36.096944 26027 tablet_bootstrap.cc:654] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.097795 26027 tablet_bootstrap.cc:492] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:36.097875 26027 ts_tablet_manager.cc:1367] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:36.098280 26027 raft_consensus.cc:348] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.098379 26027 raft_consensus.cc:374] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.098415 26027 raft_consensus.cc:729] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.098532 26027 consensus_queue.cc:260] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.098775 26027 ts_tablet_manager.cc:1398] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.098912 26027 tablet_bootstrap.cc:492] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:36.099624 26027 tablet_bootstrap.cc:654] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.100518 26027 tablet_bootstrap.cc:492] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:36.100590 26027 ts_tablet_manager.cc:1367] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:36.101029 26027 raft_consensus.cc:348] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.101112 26027 raft_consensus.cc:374] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.101147 26027 raft_consensus.cc:729] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.101276 26027 consensus_queue.cc:260] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.101521 26027 ts_tablet_manager.cc:1398] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.149009 25930 raft_consensus.cc:482] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.149137 25930 raft_consensus.cc:504] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.149547 25930 leader_election.cc:288] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:36.150005 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9405d611c5a04984bb7cfb24f25d768d" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:36.150053 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9405d611c5a04984bb7cfb24f25d768d" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:36.150131 24725 raft_consensus.cc:2454] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:36.150226 24595 raft_consensus.cc:2454] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:36.150471 24922 leader_election.cc:302] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:36.150619 25930 raft_consensus.cc:2781] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.150674 25930 raft_consensus.cc:482] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.150700 25930 raft_consensus.cc:3037] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.151517 26013 raft_consensus.cc:482] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.151623 26013 raft_consensus.cc:504] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.151679 25930 raft_consensus.cc:504] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.151989 25930 leader_election.cc:288] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:36.152019 26013 leader_election.cc:288] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:36.152283 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c35e1be84514add973014e564ad845c" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:36.152410 24595 raft_consensus.cc:2454] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:36.152421 24988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c35e1be84514add973014e564ad845c" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:36.152487 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9405d611c5a04984bb7cfb24f25d768d" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:36.152523 24988 raft_consensus.cc:2454] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:36.152577 24725 raft_consensus.cc:3037] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.152571 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9405d611c5a04984bb7cfb24f25d768d" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:36.152662 24594 raft_consensus.cc:3037] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.152719 24660 leader_election.cc:302] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:36.152917 26013 raft_consensus.cc:2781] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.152956 26013 raft_consensus.cc:482] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.152988 26013 raft_consensus.cc:3037] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.153425 24594 raft_consensus.cc:2454] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:36.153621 24725 raft_consensus.cc:2454] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:36.153735 24922 leader_election.cc:302] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:36.153928 26013 raft_consensus.cc:504] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.153996 25930 raft_consensus.cc:2781] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.154188 25930 raft_consensus.cc:686] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:36.154278 26013 leader_election.cc:288] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:36.154335 25930 consensus_queue.cc:237] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.154541 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c35e1be84514add973014e564ad845c" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:36.154666 24594 raft_consensus.cc:3037] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.154736 24988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c35e1be84514add973014e564ad845c" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:36.154836 24988 raft_consensus.cc:3037] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.155597 24594 raft_consensus.cc:2454] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:36.155998 24660 leader_election.cc:302] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:36.156159 26013 raft_consensus.cc:2781] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.156247 26013 raft_consensus.cc:686] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:36.156363 26013 consensus_queue.cc:237] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.156832 24988 raft_consensus.cc:2454] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:36.158166 24448 catalog_manager.cc:5484] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:36.158246 24445 catalog_manager.cc:5484] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:36.194006 25862 raft_consensus.cc:482] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.194135 25862 raft_consensus.cc:504] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.194525 25862 leader_election.cc:288] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:36.194787 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cf6622eed045e7a864b9e397618ecd" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:36.194859 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cf6622eed045e7a864b9e397618ecd" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:36.194919 24989 raft_consensus.cc:2454] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:36.194976 24725 raft_consensus.cc:2454] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:36.195178 24792 leader_election.cc:302] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:36.195389 25862 raft_consensus.cc:2781] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.195446 25862 raft_consensus.cc:482] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.195472 25862 raft_consensus.cc:3037] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.196359 25862 raft_consensus.cc:504] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.196664 25862 leader_election.cc:288] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:36.196959 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cf6622eed045e7a864b9e397618ecd" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:36.196959 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cf6622eed045e7a864b9e397618ecd" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:36.197098 24725 raft_consensus.cc:3037] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.197098 24989 raft_consensus.cc:3037] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.198050 24989 raft_consensus.cc:2454] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:36.198076 24725 raft_consensus.cc:2454] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:36.198309 24791 leader_election.cc:302] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:36.198511 25862 raft_consensus.cc:2781] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.198590 25862 raft_consensus.cc:686] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:36.198719 25862 consensus_queue.cc:237] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:36.200476 24445 catalog_manager.cc:5484] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
W20240501 19:56:36.339370 26036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:36.339648 26036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:36.343144 26036 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 19:56:36.440821 24989 raft_consensus.cc:1261] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.440869 24594 raft_consensus.cc:1261] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.441285 25957 consensus_queue.cc:1035] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.441624 25957 consensus_queue.cc:1035] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.444418 24594 raft_consensus.cc:1261] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.444787 24725 raft_consensus.cc:1261] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.445155 25930 consensus_queue.cc:1035] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.445281 26031 consensus_queue.cc:1035] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.464269 24989 raft_consensus.cc:1261] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.465243 24727 raft_consensus.cc:1261] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.467602 25862 consensus_queue.cc:1035] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.467988 25959 consensus_queue.cc:1035] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.468228 25986 mvcc.cc:205] Tried to move back new op lower bound from 7022974551831711744 to 7022974550649520128. Current Snapshot: MvccSnapshot[applied={T|T < 7022974551831711744}]
I20240501 19:56:36.499109 25990 mvcc.cc:205] Tried to move back new op lower bound from 7022974551925936128 to 7022974550831333376. Current Snapshot: MvccSnapshot[applied={T|T < 7022974551925936128}]
I20240501 19:56:36.502369 26070 mvcc.cc:205] Tried to move back new op lower bound from 7022974551925936128 to 7022974550831333376. Current Snapshot: MvccSnapshot[applied={T|T < 7022974551818686464}]
INSERT report
rows total: 2000
time total: 128.06 ms
time per row: 0.0640298 ms
I20240501 19:56:36.519311 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:36.525234 24838 tablet_service.cc:1467] Processing CreateTablet for tablet a4b240b461b548a087fcabdeb938df1e (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.525319 24706 tablet_service.cc:1467] Processing CreateTablet for tablet a4b240b461b548a087fcabdeb938df1e (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.525593 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b240b461b548a087fcabdeb938df1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.525628 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b240b461b548a087fcabdeb938df1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.526552 24574 tablet_service.cc:1467] Processing CreateTablet for tablet a4b240b461b548a087fcabdeb938df1e (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.526552 24576 tablet_service.cc:1467] Processing CreateTablet for tablet e996e96ce69348c09adc99dc7d9c7125 (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.526865 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b240b461b548a087fcabdeb938df1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.527474 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e996e96ce69348c09adc99dc7d9c7125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.527947 26018 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:36.528182 26021 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:36.528839 26018 tablet_bootstrap.cc:654] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.528975 26019 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:36.529029 26021 tablet_bootstrap.cc:654] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.529747 24969 tablet_service.cc:1467] Processing CreateTablet for tablet dc23e8b5ad8a4911ae766492fb823806 (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.529836 26019 tablet_bootstrap.cc:654] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.530030 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc23e8b5ad8a4911ae766492fb823806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.530934 26019 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:36.531109 24706 tablet_service.cc:1467] Processing CreateTablet for tablet dc23e8b5ad8a4911ae766492fb823806 (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.531024 26019 ts_tablet_manager.cc:1367] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:36.531344 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc23e8b5ad8a4911ae766492fb823806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.531682 26019 raft_consensus.cc:348] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.531811 26019 raft_consensus.cc:374] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.531872 26019 raft_consensus.cc:729] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.531999 26019 consensus_queue.cc:260] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.532269 26019 ts_tablet_manager.cc:1398] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:36.532431 26019 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:36.532364 24968 tablet_service.cc:1467] Processing CreateTablet for tablet e996e96ce69348c09adc99dc7d9c7125 (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.532575 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e996e96ce69348c09adc99dc7d9c7125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.533236 26019 tablet_bootstrap.cc:654] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.534366 26019 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:36.534449 26019 ts_tablet_manager.cc:1367] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:36.534621 26027 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:36.534894 26019 raft_consensus.cc:348] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.535007 26019 raft_consensus.cc:374] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.535055 26019 raft_consensus.cc:729] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.535171 26019 consensus_queue.cc:260] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.535465 26019 ts_tablet_manager.cc:1398] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.535514 26027 tablet_bootstrap.cc:654] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.536254 26018 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:36.536378 26018 ts_tablet_manager.cc:1367] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:36.536553 24838 tablet_service.cc:1467] Processing CreateTablet for tablet dc23e8b5ad8a4911ae766492fb823806 (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.536553 24836 tablet_service.cc:1467] Processing CreateTablet for tablet e996e96ce69348c09adc99dc7d9c7125 (DEFAULT_TABLE table=rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.536866 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc23e8b5ad8a4911ae766492fb823806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.536923 26018 raft_consensus.cc:348] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.537003 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e996e96ce69348c09adc99dc7d9c7125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.537027 26018 raft_consensus.cc:374] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.537058 26018 raft_consensus.cc:729] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.537164 26018 consensus_queue.cc:260] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.537428 26018 ts_tablet_manager.cc:1398] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.537573 26021 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:36.537581 26018 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:36.537650 26021 ts_tablet_manager.cc:1367] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:36.538086 26021 raft_consensus.cc:348] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.538177 26021 raft_consensus.cc:374] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.538205 26021 raft_consensus.cc:729] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.538331 26021 consensus_queue.cc:260] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.538488 26018 tablet_bootstrap.cc:654] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.538617 26021 ts_tablet_manager.cc:1398] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.539664 26018 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:36.539747 26018 ts_tablet_manager.cc:1367] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:36.540153 26018 raft_consensus.cc:348] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.540241 26018 raft_consensus.cc:374] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.540272 26018 raft_consensus.cc:729] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.540378 26018 consensus_queue.cc:260] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.540700 26018 ts_tablet_manager.cc:1398] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.540848 26027 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:36.540958 26027 ts_tablet_manager.cc:1367] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:36.541410 26027 raft_consensus.cc:348] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.541522 26027 raft_consensus.cc:374] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.541555 26027 raft_consensus.cc:729] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.541702 26027 consensus_queue.cc:260] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.541940 26027 ts_tablet_manager.cc:1398] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.542091 26027 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:36.542924 26027 tablet_bootstrap.cc:654] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.543466 26021 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:36.544420 26027 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:36.544472 26021 tablet_bootstrap.cc:654] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.544504 26027 ts_tablet_manager.cc:1367] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:36.544973 26027 raft_consensus.cc:348] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.545082 26027 raft_consensus.cc:374] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.545121 26027 raft_consensus.cc:729] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.545239 26027 consensus_queue.cc:260] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.545529 26027 ts_tablet_manager.cc:1398] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:36.546402 26021 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:36.546506 26021 ts_tablet_manager.cc:1367] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:36.547226 26021 raft_consensus.cc:348] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.547365 26021 raft_consensus.cc:374] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.547452 26021 raft_consensus.cc:729] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.547626 26021 consensus_queue.cc:260] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.548353 26021 ts_tablet_manager.cc:1398] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:36.548528 26021 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:36.549697 26021 tablet_bootstrap.cc:654] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.550671 26021 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:36.550755 26021 ts_tablet_manager.cc:1367] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:36.551084 26021 raft_consensus.cc:348] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.551167 26021 raft_consensus.cc:374] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.551187 26021 raft_consensus.cc:729] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.551290 26021 consensus_queue.cc:260] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.551527 26021 ts_tablet_manager.cc:1398] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.648367 26031 raft_consensus.cc:482] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.648488 26031 raft_consensus.cc:504] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.648924 26031 leader_election.cc:288] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:36.649217 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc23e8b5ad8a4911ae766492fb823806" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:36.649312 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc23e8b5ad8a4911ae766492fb823806" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:36.649360 24857 raft_consensus.cc:2454] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:36.649461 24725 raft_consensus.cc:2454] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:36.650079 24923 leader_election.cc:302] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:36.650260 26031 raft_consensus.cc:2781] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.650315 26031 raft_consensus.cc:482] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.650342 26031 raft_consensus.cc:3037] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.651268 26031 raft_consensus.cc:504] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.651578 26031 leader_election.cc:288] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:36.651877 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc23e8b5ad8a4911ae766492fb823806" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:36.651921 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc23e8b5ad8a4911ae766492fb823806" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:36.652007 24857 raft_consensus.cc:3037] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.652025 24725 raft_consensus.cc:3037] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.652732 24857 raft_consensus.cc:2454] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:36.652799 24725 raft_consensus.cc:2454] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:36.653056 24923 leader_election.cc:302] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:36.653223 26031 raft_consensus.cc:2781] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.653307 26031 raft_consensus.cc:686] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:36.653447 26031 consensus_queue.cc:237] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:36.655253 24448 catalog_manager.cc:5484] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:36.724563 25929 raft_consensus.cc:482] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.724705 25929 raft_consensus.cc:504] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.725133 25929 leader_election.cc:288] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:36.725577 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e996e96ce69348c09adc99dc7d9c7125" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:36.725628 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e996e96ce69348c09adc99dc7d9c7125" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:36.725720 24857 raft_consensus.cc:2454] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:36.725780 24989 raft_consensus.cc:2454] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:36.726033 24530 leader_election.cc:302] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:36.726213 25929 raft_consensus.cc:2781] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.726282 25929 raft_consensus.cc:482] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.726320 25929 raft_consensus.cc:3037] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.727219 25929 raft_consensus.cc:504] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.727511 25929 leader_election.cc:288] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:36.727839 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e996e96ce69348c09adc99dc7d9c7125" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:36.727905 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e996e96ce69348c09adc99dc7d9c7125" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:36.727957 24857 raft_consensus.cc:3037] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.728020 24989 raft_consensus.cc:3037] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.728700 24857 raft_consensus.cc:2454] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:36.728786 24989 raft_consensus.cc:2454] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:36.728979 24530 leader_election.cc:302] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:36.729153 25929 raft_consensus.cc:2781] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.729233 25929 raft_consensus.cc:686] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:36.729372 25929 consensus_queue.cc:237] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:36.731083 24448 catalog_manager.cc:5484] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:36.783776 25727 raft_consensus.cc:482] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.783895 25727 raft_consensus.cc:504] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.784294 25727 leader_election.cc:288] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:36.784581 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b240b461b548a087fcabdeb938df1e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:36.784668 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b240b461b548a087fcabdeb938df1e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:36.784716 24856 raft_consensus.cc:2454] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:36.784782 24595 raft_consensus.cc:2454] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:36.785046 24661 leader_election.cc:302] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:36.785259 25727 raft_consensus.cc:2781] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.785305 25727 raft_consensus.cc:482] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.785346 25727 raft_consensus.cc:3037] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.786213 25727 raft_consensus.cc:504] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.786537 25727 leader_election.cc:288] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:36.786840 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b240b461b548a087fcabdeb938df1e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:36.786860 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b240b461b548a087fcabdeb938df1e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:36.786960 24856 raft_consensus.cc:3037] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.786985 24595 raft_consensus.cc:3037] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.787899 24856 raft_consensus.cc:2454] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:36.787995 24595 raft_consensus.cc:2454] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:36.788146 24661 leader_election.cc:302] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:36.788296 25727 raft_consensus.cc:2781] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.788400 25727 raft_consensus.cc:686] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:36.788527 25727 consensus_queue.cc:237] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:36.790120 24448 catalog_manager.cc:5484] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
W20240501 19:56:36.934499 26086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:36.934762 26086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:36.938279 26086 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 19:56:37.030644 24595 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.031085 24856 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.031124 25727 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.031865 25727 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.033282 24725 raft_consensus.cc:1261] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.033339 24856 raft_consensus.cc:1261] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.034582 25930 consensus_queue.cc:1035] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.035686 25930 consensus_queue.cc:1035] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.053865 26127 mvcc.cc:205] Tried to move back new op lower bound from 7022974554233606144 to 7022974553246810112. Current Snapshot: MvccSnapshot[applied={T|T < 7022974554233606144}]
I20240501 19:56:37.059764 24989 raft_consensus.cc:1261] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.060189 25928 consensus_queue.cc:1035] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.065129 24856 raft_consensus.cc:1261] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.065528 25928 consensus_queue.cc:1035] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.644 ms
time per row: 0.0623221 ms
I20240501 19:56:37.104840 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:37.110194 24576 tablet_service.cc:1467] Processing CreateTablet for tablet bacb09466dc54a5a9286987d1b27d0c0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.110277 24706 tablet_service.cc:1467] Processing CreateTablet for tablet bacb09466dc54a5a9286987d1b27d0c0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.110539 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bacb09466dc54a5a9286987d1b27d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.110599 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bacb09466dc54a5a9286987d1b27d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.111397 24838 tablet_service.cc:1467] Processing CreateTablet for tablet cfd153394cac410093c6b2f057c3b0de (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.111698 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd153394cac410093c6b2f057c3b0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.112205 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5b793a014b4d84ba7dd679623fe458 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.112429 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5b793a014b4d84ba7dd679623fe458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.113281 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5b793a014b4d84ba7dd679623fe458 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.113325 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5b793a014b4d84ba7dd679623fe458 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.113559 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5b793a014b4d84ba7dd679623fe458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.113562 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5b793a014b4d84ba7dd679623fe458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.111397 24836 tablet_service.cc:1467] Processing CreateTablet for tablet bacb09466dc54a5a9286987d1b27d0c0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.113924 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bacb09466dc54a5a9286987d1b27d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.114943 26147 tablet_bootstrap.cc:492] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:37.115653 24707 tablet_service.cc:1467] Processing CreateTablet for tablet cfd153394cac410093c6b2f057c3b0de (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.115890 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd153394cac410093c6b2f057c3b0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.115949 26148 tablet_bootstrap.cc:492] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:37.116637 26147 tablet_bootstrap.cc:654] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.116830 26148 tablet_bootstrap.cc:654] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.117003 26149 tablet_bootstrap.cc:492] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:37.117900 26147 tablet_bootstrap.cc:492] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:37.117935 26148 tablet_bootstrap.cc:492] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:37.117980 26147 ts_tablet_manager.cc:1367] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:37.118000 26149 tablet_bootstrap.cc:654] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.118017 26148 ts_tablet_manager.cc:1367] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:37.118021 24969 tablet_service.cc:1467] Processing CreateTablet for tablet cfd153394cac410093c6b2f057c3b0de (DEFAULT_TABLE table=rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.118249 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd153394cac410093c6b2f057c3b0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.118409 26147 raft_consensus.cc:348] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.118525 26147 raft_consensus.cc:374] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.118477 26148 raft_consensus.cc:348] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.118557 26147 raft_consensus.cc:729] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.118569 26148 raft_consensus.cc:374] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.118592 26148 raft_consensus.cc:729] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.118690 26147 consensus_queue.cc:260] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.118700 26148 consensus_queue.cc:260] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.118945 26147 ts_tablet_manager.cc:1398] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.118955 26148 ts_tablet_manager.cc:1398] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.119132 26147 tablet_bootstrap.cc:492] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:37.119130 26148 tablet_bootstrap.cc:492] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:37.119916 26148 tablet_bootstrap.cc:654] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.120014 26147 tablet_bootstrap.cc:654] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.121212 26148 tablet_bootstrap.cc:492] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:37.121299 26148 ts_tablet_manager.cc:1367] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.121794 26148 raft_consensus.cc:348] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.121901 26148 raft_consensus.cc:374] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.121932 26148 raft_consensus.cc:729] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.122056 26147 tablet_bootstrap.cc:492] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:37.122040 26148 consensus_queue.cc:260] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.122126 26147 ts_tablet_manager.cc:1367] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:37.122516 26148 ts_tablet_manager.cc:1398] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.122854 26146 tablet_bootstrap.cc:492] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:37.122814 26147 raft_consensus.cc:348] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.122921 26147 raft_consensus.cc:374] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.122953 26147 raft_consensus.cc:729] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.123108 26147 consensus_queue.cc:260] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.123241 26149 tablet_bootstrap.cc:492] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:37.123489 26147 ts_tablet_manager.cc:1398] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.123742 26149 ts_tablet_manager.cc:1367] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:37.123839 26146 tablet_bootstrap.cc:654] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.124195 26149 raft_consensus.cc:348] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.124323 26149 raft_consensus.cc:374] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.124356 26149 raft_consensus.cc:729] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.124511 26149 consensus_queue.cc:260] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.125223 26149 ts_tablet_manager.cc:1398] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.125423 26149 tablet_bootstrap.cc:492] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:37.125841 26146 tablet_bootstrap.cc:492] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:37.125926 26146 ts_tablet_manager.cc:1367] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:37.126348 26149 tablet_bootstrap.cc:654] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.126364 26146 raft_consensus.cc:348] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.126495 26146 raft_consensus.cc:374] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.126534 26146 raft_consensus.cc:729] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.126655 26146 consensus_queue.cc:260] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.126935 26146 ts_tablet_manager.cc:1398] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.127128 26146 tablet_bootstrap.cc:492] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:37.127547 26149 tablet_bootstrap.cc:492] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:37.127617 26149 ts_tablet_manager.cc:1367] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:37.127892 26146 tablet_bootstrap.cc:654] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.127918 26149 raft_consensus.cc:348] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.128021 26149 raft_consensus.cc:374] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.128064 26149 raft_consensus.cc:729] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.128185 26149 consensus_queue.cc:260] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.128449 26149 ts_tablet_manager.cc:1398] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.129140 26146 tablet_bootstrap.cc:492] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:37.129220 26146 ts_tablet_manager.cc:1367] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.129617 26146 raft_consensus.cc:348] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.129757 26146 raft_consensus.cc:374] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.129788 26146 raft_consensus.cc:729] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.130089 26146 consensus_queue.cc:260] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.130360 26146 ts_tablet_manager.cc:1398] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.130494 26146 tablet_bootstrap.cc:492] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:37.132103 26146 tablet_bootstrap.cc:654] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.133039 26146 tablet_bootstrap.cc:492] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:37.133102 26146 ts_tablet_manager.cc:1367] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:37.133388 26146 raft_consensus.cc:348] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.133467 26146 raft_consensus.cc:374] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.133497 26146 raft_consensus.cc:729] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.133576 26146 consensus_queue.cc:260] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.133790 26146 ts_tablet_manager.cc:1398] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.190011 25930 raft_consensus.cc:482] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.190141 25930 raft_consensus.cc:504] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.190596 25930 leader_election.cc:288] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:37.191025 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd153394cac410093c6b2f057c3b0de" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:37.191069 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd153394cac410093c6b2f057c3b0de" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:37.191190 24856 raft_consensus.cc:2454] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:37.191229 24725 raft_consensus.cc:2454] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:37.191475 24923 leader_election.cc:302] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:37.191625 25930 raft_consensus.cc:2781] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.191694 25930 raft_consensus.cc:482] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.191738 25930 raft_consensus.cc:3037] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.192518 25930 raft_consensus.cc:504] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.192875 25930 leader_election.cc:288] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:37.193167 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd153394cac410093c6b2f057c3b0de" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:37.193197 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd153394cac410093c6b2f057c3b0de" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:37.193271 24856 raft_consensus.cc:3037] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.193290 24725 raft_consensus.cc:3037] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.194036 24725 raft_consensus.cc:2454] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:37.194056 24856 raft_consensus.cc:2454] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:37.194300 24922 leader_election.cc:302] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:37.194437 25930 raft_consensus.cc:2781] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.194532 25930 raft_consensus.cc:686] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:37.194682 25930 consensus_queue.cc:237] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.196365 24445 catalog_manager.cc:5484] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:37.226651 25930 raft_consensus.cc:482] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.226821 25930 raft_consensus.cc:504] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.227272 25930 leader_election.cc:288] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:37.227805 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5b793a014b4d84ba7dd679623fe458" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:37.227805 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5b793a014b4d84ba7dd679623fe458" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:37.227955 24595 raft_consensus.cc:2454] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:37.227955 24725 raft_consensus.cc:2454] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:37.228255 24922 leader_election.cc:302] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:37.228420 26031 raft_consensus.cc:2781] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.228489 26031 raft_consensus.cc:482] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.228519 26031 raft_consensus.cc:3037] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.229437 26031 raft_consensus.cc:504] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.229733 26031 leader_election.cc:288] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:37.230051 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5b793a014b4d84ba7dd679623fe458" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:37.230082 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5b793a014b4d84ba7dd679623fe458" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:37.230186 24725 raft_consensus.cc:3037] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.230190 24595 raft_consensus.cc:3037] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.231077 24725 raft_consensus.cc:2454] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:37.231078 24595 raft_consensus.cc:2454] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:37.231325 24922 leader_election.cc:302] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:37.231520 26031 raft_consensus.cc:2781] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.231606 26031 raft_consensus.cc:686] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:37.231745 26031 consensus_queue.cc:237] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.233466 24445 catalog_manager.cc:5484] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:37.322708 25727 raft_consensus.cc:482] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.322846 25727 raft_consensus.cc:504] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.323288 25727 leader_election.cc:288] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:37.323663 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacb09466dc54a5a9286987d1b27d0c0" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:37.323706 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacb09466dc54a5a9286987d1b27d0c0" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:37.323792 24856 raft_consensus.cc:2454] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:37.323863 24595 raft_consensus.cc:2454] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:37.324069 24661 leader_election.cc:302] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:37.324295 25727 raft_consensus.cc:2781] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.324343 25727 raft_consensus.cc:482] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.324373 25727 raft_consensus.cc:3037] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.325201 25727 raft_consensus.cc:504] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.325503 25727 leader_election.cc:288] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:37.325796 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacb09466dc54a5a9286987d1b27d0c0" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:37.325794 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacb09466dc54a5a9286987d1b27d0c0" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:37.325887 24595 raft_consensus.cc:3037] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.325894 24856 raft_consensus.cc:3037] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.326584 24856 raft_consensus.cc:2454] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:37.326608 24595 raft_consensus.cc:2454] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:37.326835 24661 leader_election.cc:302] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:37.327026 25727 raft_consensus.cc:2781] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.327137 25727 raft_consensus.cc:686] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:37.327286 25727 consensus_queue.cc:237] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:37.329391 24445 catalog_manager.cc:5484] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
W20240501 19:56:37.459250 26162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.459529 26162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.462575 26162 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 19:56:37.546551 24725 raft_consensus.cc:1261] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.546630 24594 raft_consensus.cc:1261] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.547104 26031 consensus_queue.cc:1035] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.547992 24594 raft_consensus.cc:1261] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.548167 26031 consensus_queue.cc:1035] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.548426 26013 consensus_queue.cc:1035] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.548946 24856 raft_consensus.cc:1261] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.551586 26013 consensus_queue.cc:1035] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.562430 24857 raft_consensus.cc:1261] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.563005 26031 consensus_queue.cc:1035] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.569591 24725 raft_consensus.cc:1261] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.570183 26031 consensus_queue.cc:1035] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.709 ms
time per row: 0.0593545 ms
I20240501 19:56:37.625685 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:37.631122 24574 tablet_service.cc:1467] Processing CreateTablet for tablet c3d31791486b4e68a9e5283faadb3f8c (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.631350 24836 tablet_service.cc:1467] Processing CreateTablet for tablet c3d31791486b4e68a9e5283faadb3f8c (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.631384 24969 tablet_service.cc:1467] Processing CreateTablet for tablet c3d31791486b4e68a9e5283faadb3f8c (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.631467 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d31791486b4e68a9e5283faadb3f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.631507 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 3b692b5b96534e79a32a8b06b92f32d6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.631639 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d31791486b4e68a9e5283faadb3f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.631675 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d31791486b4e68a9e5283faadb3f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.632009 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b692b5b96534e79a32a8b06b92f32d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.634263 26205 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:37.634624 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 3b692b5b96534e79a32a8b06b92f32d6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.634871 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b692b5b96534e79a32a8b06b92f32d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.635069 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 3b692b5b96534e79a32a8b06b92f32d6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.635105 24576 tablet_service.cc:1467] Processing CreateTablet for tablet f89e81297b66402ea09bdb415fb7f385 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.635249 24707 tablet_service.cc:1467] Processing CreateTablet for tablet f89e81297b66402ea09bdb415fb7f385 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.635332 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89e81297b66402ea09bdb415fb7f385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.635462 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b692b5b96534e79a32a8b06b92f32d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.635502 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89e81297b66402ea09bdb415fb7f385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.637216 24838 tablet_service.cc:1467] Processing CreateTablet for tablet f89e81297b66402ea09bdb415fb7f385 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.637446 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89e81297b66402ea09bdb415fb7f385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.637603 26208 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:37.637703 26205 tablet_bootstrap.cc:654] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.637879 26207 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:37.638345 26206 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:37.638741 26207 tablet_bootstrap.cc:654] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.638793 26208 tablet_bootstrap.cc:654] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.639250 26206 tablet_bootstrap.cc:654] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.639920 26207 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:37.640010 26207 ts_tablet_manager.cc:1367] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:37.640053 26208 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:37.640062 26205 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:37.640143 26208 ts_tablet_manager.cc:1367] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:37.640143 26205 ts_tablet_manager.cc:1367] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:37.640417 26207 raft_consensus.cc:348] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.640511 26207 raft_consensus.cc:374] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.640548 26207 raft_consensus.cc:729] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.640580 26208 raft_consensus.cc:348] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.640580 26205 raft_consensus.cc:348] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.640692 26205 raft_consensus.cc:374] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.640692 26208 raft_consensus.cc:374] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.640679 26207 consensus_queue.cc:260] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.640738 26208 raft_consensus.cc:729] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.640795 26205 raft_consensus.cc:729] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.640983 26207 ts_tablet_manager.cc:1398] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.640952 26205 consensus_queue.cc:260] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.641158 26207 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:37.641232 26205 ts_tablet_manager.cc:1398] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.641758 26208 consensus_queue.cc:260] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.641850 26205 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:37.641911 26207 tablet_bootstrap.cc:654] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.642716 26205 tablet_bootstrap.cc:654] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.643287 26207 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:37.643355 26207 ts_tablet_manager.cc:1367] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.643730 26207 raft_consensus.cc:348] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.643811 26207 raft_consensus.cc:374] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.643844 26207 raft_consensus.cc:729] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.643947 26207 consensus_queue.cc:260] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.644208 26207 ts_tablet_manager.cc:1398] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:37.644310 26206 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:37.644354 26207 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:37.644418 26206 ts_tablet_manager.cc:1367] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:37.644984 26206 raft_consensus.cc:348] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.645092 26206 raft_consensus.cc:374] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.645141 26207 tablet_bootstrap.cc:654] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.645143 26206 raft_consensus.cc:729] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.645195 26208 ts_tablet_manager.cc:1398] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:37.645292 26206 consensus_queue.cc:260] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.645637 26206 ts_tablet_manager.cc:1398] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.645887 26206 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:37.646282 26207 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:37.646358 26207 ts_tablet_manager.cc:1367] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:37.646831 26207 raft_consensus.cc:348] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.646944 26207 raft_consensus.cc:374] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.646983 26207 raft_consensus.cc:729] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.647145 26207 consensus_queue.cc:260] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.647392 26207 ts_tablet_manager.cc:1398] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.648366 26205 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:37.648487 26205 ts_tablet_manager.cc:1367] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:37.648936 26205 raft_consensus.cc:348] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.649013 26205 raft_consensus.cc:374] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.649032 26205 raft_consensus.cc:729] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.649067 26206 tablet_bootstrap.cc:654] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.649137 26205 consensus_queue.cc:260] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.649446 26205 ts_tablet_manager.cc:1398] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.650068 26206 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:37.650147 26206 ts_tablet_manager.cc:1367] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:37.650539 26206 raft_consensus.cc:348] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.650624 26206 raft_consensus.cc:374] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.650673 26206 raft_consensus.cc:729] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.650799 26206 consensus_queue.cc:260] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.651108 26206 ts_tablet_manager.cc:1398] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.651293 26206 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:37.652089 26206 tablet_bootstrap.cc:654] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.653371 26206 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:37.653477 26206 ts_tablet_manager.cc:1367] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.653930 26206 raft_consensus.cc:348] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.654034 26206 raft_consensus.cc:374] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.654079 26206 raft_consensus.cc:729] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.654204 26206 consensus_queue.cc:260] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.654474 26206 ts_tablet_manager.cc:1398] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.804031 25929 raft_consensus.cc:482] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.804143 25929 raft_consensus.cc:504] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.804536 25929 leader_election.cc:288] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:37.804899 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d31791486b4e68a9e5283faadb3f8c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:37.805050 24856 raft_consensus.cc:2454] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:37.805011 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d31791486b4e68a9e5283faadb3f8c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:37.805169 24989 raft_consensus.cc:2454] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:37.805373 24530 leader_election.cc:302] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:37.805553 25929 raft_consensus.cc:2781] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.805609 25929 raft_consensus.cc:482] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.805653 25929 raft_consensus.cc:3037] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.806434 25929 raft_consensus.cc:504] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.806802 25929 leader_election.cc:288] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:37.807108 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d31791486b4e68a9e5283faadb3f8c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:37.807221 24856 raft_consensus.cc:3037] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.807217 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d31791486b4e68a9e5283faadb3f8c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:37.807333 24989 raft_consensus.cc:3037] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.808254 24856 raft_consensus.cc:2454] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:37.808254 24989 raft_consensus.cc:2454] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:37.808538 24530 leader_election.cc:302] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:37.808718 25929 raft_consensus.cc:2781] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.808808 25929 raft_consensus.cc:686] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:37.808952 25929 consensus_queue.cc:237] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:37.810770 24446 catalog_manager.cc:5484] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:37.828334 26197 raft_consensus.cc:482] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.828442 26197 raft_consensus.cc:504] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.828830 26197 leader_election.cc:288] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:37.829144 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b692b5b96534e79a32a8b06b92f32d6" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:37.829188 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b692b5b96534e79a32a8b06b92f32d6" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:37.829278 24594 raft_consensus.cc:2454] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:37.829350 24989 raft_consensus.cc:2454] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:37.829553 24791 leader_election.cc:302] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:37.829708 26197 raft_consensus.cc:2781] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.829769 26197 raft_consensus.cc:482] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.829830 26197 raft_consensus.cc:3037] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.830682 26197 raft_consensus.cc:504] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.831018 26197 leader_election.cc:288] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:37.831329 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b692b5b96534e79a32a8b06b92f32d6" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:37.831431 24594 raft_consensus.cc:3037] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.831362 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b692b5b96534e79a32a8b06b92f32d6" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:37.831521 24989 raft_consensus.cc:3037] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.832260 24594 raft_consensus.cc:2454] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:37.832669 24791 leader_election.cc:302] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:37.832724 24989 raft_consensus.cc:2454] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:37.832891 26197 raft_consensus.cc:2781] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.832975 26197 raft_consensus.cc:686] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:37.833104 26197 consensus_queue.cc:237] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:37.835062 24446 catalog_manager.cc:5484] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:37.916147 25929 raft_consensus.cc:482] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.916270 25929 raft_consensus.cc:504] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.916692 25929 leader_election.cc:288] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:37.917066 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f89e81297b66402ea09bdb415fb7f385" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:37.917066 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f89e81297b66402ea09bdb415fb7f385" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:37.917209 24725 raft_consensus.cc:2454] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:37.917209 24856 raft_consensus.cc:2454] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:37.917539 24530 leader_election.cc:302] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:37.917726 25929 raft_consensus.cc:2781] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.917807 25929 raft_consensus.cc:482] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.917862 25929 raft_consensus.cc:3037] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.919209 25929 raft_consensus.cc:504] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.919737 25929 leader_election.cc:288] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:37.920049 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f89e81297b66402ea09bdb415fb7f385" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:37.920075 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f89e81297b66402ea09bdb415fb7f385" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:37.920157 24725 raft_consensus.cc:3037] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.920214 24856 raft_consensus.cc:3037] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.921006 24725 raft_consensus.cc:2454] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:37.921277 24529 leader_election.cc:302] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:37.921375 24856 raft_consensus.cc:2454] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:37.921676 25929 raft_consensus.cc:2781] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.921797 25929 raft_consensus.cc:686] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:37.921985 25929 consensus_queue.cc:237] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:37.923960 24446 catalog_manager.cc:5484] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
W20240501 19:56:38.046137 26221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:38.046397 26221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:38.049386 26221 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 19:56:38.142554 24989 raft_consensus.cc:1261] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.142709 24594 raft_consensus.cc:1261] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.142954 26197 consensus_queue.cc:1035] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.144201 26197 consensus_queue.cc:1035] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.146312 24988 raft_consensus.cc:1261] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.146414 24856 raft_consensus.cc:1261] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.146853 26145 consensus_queue.cc:1035] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.149094 25929 consensus_queue.cc:1035] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.149830 24725 raft_consensus.cc:1261] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.150272 25929 consensus_queue.cc:1035] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.151710 24857 raft_consensus.cc:1261] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.154143 26145 consensus_queue.cc:1035] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.717 ms
time per row: 0.0573584 ms
I20240501 19:56:38.210893 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:38.216648 24707 tablet_service.cc:1467] Processing CreateTablet for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.216828 24574 tablet_service.cc:1467] Processing CreateTablet for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.216908 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 2b02546b0c8c404c919ae9998984188f (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.216996 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.217126 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b02546b0c8c404c919ae9998984188f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.217625 24968 tablet_service.cc:1467] Processing CreateTablet for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.217882 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.217839 24969 tablet_service.cc:1467] Processing CreateTablet for tablet dd88b75ba5ff41cdbe02f2b83d949681 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.218063 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd88b75ba5ff41cdbe02f2b83d949681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.219275 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 2b02546b0c8c404c919ae9998984188f (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.219484 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b02546b0c8c404c919ae9998984188f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.220103 26279 tablet_bootstrap.cc:492] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:38.220208 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 2b02546b0c8c404c919ae9998984188f (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.216648 24705 tablet_service.cc:1467] Processing CreateTablet for tablet dd88b75ba5ff41cdbe02f2b83d949681 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.220403 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b02546b0c8c404c919ae9998984188f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.220530 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd88b75ba5ff41cdbe02f2b83d949681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.221174 26279 tablet_bootstrap.cc:654] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.222383 26279 tablet_bootstrap.cc:492] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:38.222457 26279 ts_tablet_manager.cc:1367] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:38.222528 24836 tablet_service.cc:1467] Processing CreateTablet for tablet dd88b75ba5ff41cdbe02f2b83d949681 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.222788 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd88b75ba5ff41cdbe02f2b83d949681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.222900 26279 raft_consensus.cc:348] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.222991 26279 raft_consensus.cc:374] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.223023 26279 raft_consensus.cc:729] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.223201 26279 consensus_queue.cc:260] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.223778 26278 tablet_bootstrap.cc:492] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:38.223990 26279 ts_tablet_manager.cc:1398] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.224138 26279 tablet_bootstrap.cc:492] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:38.224551 26278 tablet_bootstrap.cc:654] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.224959 26281 tablet_bootstrap.cc:492] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:38.224982 26279 tablet_bootstrap.cc:654] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.225530 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.226006 26281 tablet_bootstrap.cc:654] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.226245 26279 tablet_bootstrap.cc:492] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:38.226327 26279 ts_tablet_manager.cc:1367] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:38.226796 26279 raft_consensus.cc:348] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.226898 26279 raft_consensus.cc:374] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.226926 26279 raft_consensus.cc:729] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.226975 26281 tablet_bootstrap.cc:492] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:38.227111 26281 ts_tablet_manager.cc:1367] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:38.227131 26279 consensus_queue.cc:260] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.227419 26279 ts_tablet_manager.cc:1398] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.227591 26279 tablet_bootstrap.cc:492] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:38.227610 26281 raft_consensus.cc:348] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.227730 26281 raft_consensus.cc:374] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.227785 26281 raft_consensus.cc:729] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.227912 26281 consensus_queue.cc:260] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.228253 26281 ts_tablet_manager.cc:1398] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.228428 26279 tablet_bootstrap.cc:654] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.229097 26277 tablet_bootstrap.cc:492] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:38.229979 26277 tablet_bootstrap.cc:654] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.230170 26279 tablet_bootstrap.cc:492] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:38.230280 26279 ts_tablet_manager.cc:1367] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:38.230685 26279 raft_consensus.cc:348] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.230820 26279 raft_consensus.cc:374] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.230882 26279 raft_consensus.cc:729] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.231093 26279 consensus_queue.cc:260] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.231529 26279 ts_tablet_manager.cc:1398] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.231822 26277 tablet_bootstrap.cc:492] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:38.231899 26277 ts_tablet_manager.cc:1367] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:38.232254 26277 raft_consensus.cc:348] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.232340 26277 raft_consensus.cc:374] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.232374 26277 raft_consensus.cc:729] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.232494 26277 consensus_queue.cc:260] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.232753 26277 ts_tablet_manager.cc:1398] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:38.232918 26277 tablet_bootstrap.cc:492] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:38.233736 26277 tablet_bootstrap.cc:654] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.234735 26277 tablet_bootstrap.cc:492] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:38.234817 26277 ts_tablet_manager.cc:1367] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.235224 26277 raft_consensus.cc:348] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.235319 26277 raft_consensus.cc:374] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.235355 26277 raft_consensus.cc:729] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.235457 26277 consensus_queue.cc:260] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.235492 26278 tablet_bootstrap.cc:492] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:38.235576 26278 ts_tablet_manager.cc:1367] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:38.235718 26277 ts_tablet_manager.cc:1398] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.235823 26277 tablet_bootstrap.cc:492] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:38.235992 26278 raft_consensus.cc:348] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.236068 26278 raft_consensus.cc:374] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.236088 26278 raft_consensus.cc:729] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.236176 26278 consensus_queue.cc:260] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.236409 26278 ts_tablet_manager.cc:1398] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:38.236529 26277 tablet_bootstrap.cc:654] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.236595 26278 tablet_bootstrap.cc:492] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:38.237488 26278 tablet_bootstrap.cc:654] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.237643 26277 tablet_bootstrap.cc:492] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:38.237730 26277 ts_tablet_manager.cc:1367] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.238123 26277 raft_consensus.cc:348] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.238237 26277 raft_consensus.cc:374] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.238286 26277 raft_consensus.cc:729] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.238404 26277 consensus_queue.cc:260] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.238534 26278 tablet_bootstrap.cc:492] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:38.238615 26278 ts_tablet_manager.cc:1367] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:38.238811 26277 ts_tablet_manager.cc:1398] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.239037 26278 raft_consensus.cc:348] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.239181 26278 raft_consensus.cc:374] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.239218 26278 raft_consensus.cc:729] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.239468 26278 consensus_queue.cc:260] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.239746 26278 ts_tablet_manager.cc:1398] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.300176 26258 raft_consensus.cc:482] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.300312 26258 raft_consensus.cc:504] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.300717 26258 leader_election.cc:288] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:38.301049 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b02546b0c8c404c919ae9998984188f" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:38.301062 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b02546b0c8c404c919ae9998984188f" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:38.301168 24725 raft_consensus.cc:2454] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:38.301198 24989 raft_consensus.cc:2454] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:38.301476 24529 leader_election.cc:302] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:38.301661 26258 raft_consensus.cc:2781] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.301724 26258 raft_consensus.cc:482] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.301751 26258 raft_consensus.cc:3037] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.302572 26258 raft_consensus.cc:504] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.302865 26258 leader_election.cc:288] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:38.303143 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b02546b0c8c404c919ae9998984188f" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:38.303166 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b02546b0c8c404c919ae9998984188f" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:38.303239 24725 raft_consensus.cc:3037] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.303254 24989 raft_consensus.cc:3037] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.303957 24989 raft_consensus.cc:2454] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:38.303968 24725 raft_consensus.cc:2454] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:38.304208 24530 leader_election.cc:302] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:38.304349 26258 raft_consensus.cc:2781] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.304426 26258 raft_consensus.cc:686] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:38.304584 26258 consensus_queue.cc:237] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.306483 24454 catalog_manager.cc:5484] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:38.497287 26013 raft_consensus.cc:482] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.497421 26013 raft_consensus.cc:504] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.497830 26013 leader_election.cc:288] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:38.498183 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2353c8b5f284fd3bd1c5f25e2eabe8f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:38.498207 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2353c8b5f284fd3bd1c5f25e2eabe8f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:38.498327 24989 raft_consensus.cc:2454] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:38.498327 24595 raft_consensus.cc:2454] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:38.498569 24661 leader_election.cc:302] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:38.498762 26013 raft_consensus.cc:2781] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.498826 26013 raft_consensus.cc:482] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.498878 26013 raft_consensus.cc:3037] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.499763 26013 raft_consensus.cc:504] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.500073 26013 leader_election.cc:288] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:38.500460 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2353c8b5f284fd3bd1c5f25e2eabe8f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:38.500499 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2353c8b5f284fd3bd1c5f25e2eabe8f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:38.500613 24595 raft_consensus.cc:3037] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.500613 24989 raft_consensus.cc:3037] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.501557 24989 raft_consensus.cc:2454] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:38.501557 24595 raft_consensus.cc:2454] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:38.501839 24661 leader_election.cc:302] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:38.502066 26013 raft_consensus.cc:2781] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.502274 26013 raft_consensus.cc:686] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:38.502419 26013 consensus_queue.cc:237] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.504076 24454 catalog_manager.cc:5484] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:38.525300 26013 raft_consensus.cc:482] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.525429 26013 raft_consensus.cc:504] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.525748 26013 leader_election.cc:288] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:38.526144 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd88b75ba5ff41cdbe02f2b83d949681" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:38.526144 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd88b75ba5ff41cdbe02f2b83d949681" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:38.526294 24989 raft_consensus.cc:2454] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:38.526294 24856 raft_consensus.cc:2454] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:38.526597 24661 leader_election.cc:302] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:38.526764 26013 raft_consensus.cc:2781] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.526824 26013 raft_consensus.cc:482] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.526872 26013 raft_consensus.cc:3037] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.527791 26013 raft_consensus.cc:504] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.528141 26013 leader_election.cc:288] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:38.528483 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd88b75ba5ff41cdbe02f2b83d949681" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:38.528492 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd88b75ba5ff41cdbe02f2b83d949681" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:38.528584 24857 raft_consensus.cc:3037] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.528617 24989 raft_consensus.cc:3037] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.529318 24857 raft_consensus.cc:2454] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:38.529368 24989 raft_consensus.cc:2454] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:38.529772 24661 leader_election.cc:302] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:38.530061 26013 raft_consensus.cc:2781] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.530148 26013 raft_consensus.cc:686] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:38.530287 26013 consensus_queue.cc:237] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.532161 24454 catalog_manager.cc:5484] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
W20240501 19:56:38.698359 26294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:38.698585 26294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:38.701875 26294 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 19:56:38.786172 26258 consensus_queue.cc:1035] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.788080 24988 raft_consensus.cc:1261] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.788184 24989 raft_consensus.cc:1261] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.788617 24595 raft_consensus.cc:1261] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.789122 26291 consensus_queue.cc:1035] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.789438 26291 consensus_queue.cc:1035] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.790796 26145 consensus_queue.cc:1035] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.792023 24857 raft_consensus.cc:1261] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.792445 26291 consensus_queue.cc:1035] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.793787 24988 raft_consensus.cc:1261] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.795387 26291 consensus_queue.cc:1035] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.793 ms
time per row: 0.0548964 ms
I20240501 19:56:38.853598 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:38.858770 24836 tablet_service.cc:1467] Processing CreateTablet for tablet cfb037c9bf1a4ed080745df687ac4a89 (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.859120 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb037c9bf1a4ed080745df687ac4a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.859122 24574 tablet_service.cc:1467] Processing CreateTablet for tablet cfb037c9bf1a4ed080745df687ac4a89 (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.859364 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb037c9bf1a4ed080745df687ac4a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.860080 24967 tablet_service.cc:1467] Processing CreateTablet for tablet f1cd92f84a944b94b5948123e3e1d7a1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.860391 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cd92f84a944b94b5948123e3e1d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.860440 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 53981ad3756d4c54994f7aea973c5cde (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.860675 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53981ad3756d4c54994f7aea973c5cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.861649 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 53981ad3756d4c54994f7aea973c5cde (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.861866 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53981ad3756d4c54994f7aea973c5cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.861867 24836 tablet_service.cc:1467] Processing CreateTablet for tablet f1cd92f84a944b94b5948123e3e1d7a1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.862058 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cd92f84a944b94b5948123e3e1d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.862869 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 53981ad3756d4c54994f7aea973c5cde (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.863134 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53981ad3756d4c54994f7aea973c5cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.864053 26354 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:38.864253 24707 tablet_service.cc:1467] Processing CreateTablet for tablet cfb037c9bf1a4ed080745df687ac4a89 (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.864506 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb037c9bf1a4ed080745df687ac4a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.864957 26354 tablet_bootstrap.cc:654] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.866564 26354 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:38.866494 26355 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:38.866660 26354 ts_tablet_manager.cc:1367] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:38.866755 24706 tablet_service.cc:1467] Processing CreateTablet for tablet f1cd92f84a944b94b5948123e3e1d7a1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.866977 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cd92f84a944b94b5948123e3e1d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.867231 26354 raft_consensus.cc:348] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.867350 26354 raft_consensus.cc:374] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.867383 26354 raft_consensus.cc:729] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.867502 26354 consensus_queue.cc:260] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.867610 26355 tablet_bootstrap.cc:654] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.867873 26354 ts_tablet_manager.cc:1398] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.868165 26354 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:38.869136 26354 tablet_bootstrap.cc:654] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.869406 26355 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:38.869498 26355 ts_tablet_manager.cc:1367] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:38.870085 26355 raft_consensus.cc:348] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.870201 26355 raft_consensus.cc:374] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.870240 26355 raft_consensus.cc:729] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.870322 26357 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:38.870405 26355 consensus_queue.cc:260] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.870699 26355 ts_tablet_manager.cc:1398] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.870904 26355 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:38.871107 26357 tablet_bootstrap.cc:654] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.871824 26355 tablet_bootstrap.cc:654] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.872642 26353 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:38.873242 26355 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:38.873324 26355 ts_tablet_manager.cc:1367] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.873677 26353 tablet_bootstrap.cc:654] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.873827 26355 raft_consensus.cc:348] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.873939 26355 raft_consensus.cc:374] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.873975 26355 raft_consensus.cc:729] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.874085 26355 consensus_queue.cc:260] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.874377 26355 ts_tablet_manager.cc:1398] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.875331 26353 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:38.875403 26353 ts_tablet_manager.cc:1367] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:38.875717 26353 raft_consensus.cc:348] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.875800 26353 raft_consensus.cc:374] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.875821 26353 raft_consensus.cc:729] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.875977 26353 consensus_queue.cc:260] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.876263 26353 ts_tablet_manager.cc:1398] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.876881 26354 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:38.876907 26353 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:38.876965 26354 ts_tablet_manager.cc:1367] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:38.877362 26357 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:38.877447 26357 ts_tablet_manager.cc:1367] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:38.877458 26354 raft_consensus.cc:348] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.877576 26354 raft_consensus.cc:374] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.877609 26354 raft_consensus.cc:729] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.877720 26353 tablet_bootstrap.cc:654] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.877758 26354 consensus_queue.cc:260] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.878041 26354 ts_tablet_manager.cc:1398] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.878161 26357 raft_consensus.cc:348] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.878271 26357 raft_consensus.cc:374] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.878302 26357 raft_consensus.cc:729] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.878417 26357 consensus_queue.cc:260] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.879504 26357 ts_tablet_manager.cc:1398] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:38.879523 26353 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:38.879593 26353 ts_tablet_manager.cc:1367] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:38.880090 26357 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:38.880951 26353 raft_consensus.cc:348] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.881083 26353 raft_consensus.cc:374] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.881122 26353 raft_consensus.cc:729] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.881287 26357 tablet_bootstrap.cc:654] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.881250 26353 consensus_queue.cc:260] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.881578 26353 ts_tablet_manager.cc:1398] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:38.881733 26353 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:38.882541 26353 tablet_bootstrap.cc:654] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.883634 26357 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:38.883723 26357 ts_tablet_manager.cc:1367] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:38.883970 26353 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:38.884058 26353 ts_tablet_manager.cc:1367] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:38.884203 26357 raft_consensus.cc:348] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.884289 26357 raft_consensus.cc:374] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.884318 26357 raft_consensus.cc:729] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.884443 26357 consensus_queue.cc:260] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:38.884549 26353 raft_consensus.cc:348] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.884634 26353 raft_consensus.cc:374] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.884670 26353 raft_consensus.cc:729] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.884773 26353 consensus_queue.cc:260] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:38.885022 26353 ts_tablet_manager.cc:1398] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.885120 26357 ts_tablet_manager.cc:1398] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.926421 26197 raft_consensus.cc:482] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.926594 26197 raft_consensus.cc:504] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.926998 26197 leader_election.cc:288] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:38.927435 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb037c9bf1a4ed080745df687ac4a89" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:38.927455 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb037c9bf1a4ed080745df687ac4a89" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:38.927574 24595 raft_consensus.cc:2454] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:38.927577 24725 raft_consensus.cc:2454] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:38.927850 24791 leader_election.cc:302] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:38.928001 26197 raft_consensus.cc:2781] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.928068 26197 raft_consensus.cc:482] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.928120 26197 raft_consensus.cc:3037] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.928977 26197 raft_consensus.cc:504] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.929368 26197 leader_election.cc:288] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:38.929828 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb037c9bf1a4ed080745df687ac4a89" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:38.930003 24595 raft_consensus.cc:3037] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.930007 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb037c9bf1a4ed080745df687ac4a89" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:38.930109 24725 raft_consensus.cc:3037] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.930883 24595 raft_consensus.cc:2454] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:38.931061 24725 raft_consensus.cc:2454] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:38.931237 24791 leader_election.cc:302] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:38.931391 26197 raft_consensus.cc:2781] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.931597 26197 raft_consensus.cc:686] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:38.931762 26197 consensus_queue.cc:237] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:38.933714 24450 catalog_manager.cc:5484] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:39.083293 26291 raft_consensus.cc:482] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.083417 26291 raft_consensus.cc:504] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.083840 26291 leader_election.cc:288] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:39.084172 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cd92f84a944b94b5948123e3e1d7a1" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:39.084213 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cd92f84a944b94b5948123e3e1d7a1" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:39.084290 24856 raft_consensus.cc:2454] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:39.084365 24989 raft_consensus.cc:2454] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:39.084614 24661 leader_election.cc:302] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:39.084811 26291 raft_consensus.cc:2781] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.084867 26291 raft_consensus.cc:482] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.084923 26291 raft_consensus.cc:3037] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.085795 26291 raft_consensus.cc:504] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.086079 26291 leader_election.cc:288] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:39.086403 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cd92f84a944b94b5948123e3e1d7a1" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:39.086422 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cd92f84a944b94b5948123e3e1d7a1" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:39.086520 24856 raft_consensus.cc:3037] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.086555 24989 raft_consensus.cc:3037] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.087320 24989 raft_consensus.cc:2454] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:39.087383 24856 raft_consensus.cc:2454] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:39.087595 24661 leader_election.cc:302] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:39.087788 26291 raft_consensus.cc:2781] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.087881 26291 raft_consensus.cc:686] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:39.088028 26291 consensus_queue.cc:237] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.089790 24450 catalog_manager.cc:5484] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:39.132407 25930 raft_consensus.cc:482] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.132542 25930 raft_consensus.cc:504] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.132925 25930 leader_election.cc:288] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:39.133272 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53981ad3756d4c54994f7aea973c5cde" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:39.133273 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53981ad3756d4c54994f7aea973c5cde" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:39.133399 24856 raft_consensus.cc:2454] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:39.133426 24595 raft_consensus.cc:2454] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:39.133652 24923 leader_election.cc:302] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:39.133865 25930 raft_consensus.cc:2781] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.133910 25930 raft_consensus.cc:482] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.133955 25930 raft_consensus.cc:3037] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.134753 25930 raft_consensus.cc:504] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.135128 25930 leader_election.cc:288] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:39.135418 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53981ad3756d4c54994f7aea973c5cde" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:39.135430 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53981ad3756d4c54994f7aea973c5cde" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:39.135530 24595 raft_consensus.cc:3037] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.135541 24856 raft_consensus.cc:3037] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.136291 24595 raft_consensus.cc:2454] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:39.136302 24856 raft_consensus.cc:2454] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:39.136597 24923 leader_election.cc:302] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:39.136761 25930 raft_consensus.cc:2781] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.136981 25930 raft_consensus.cc:686] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:39.137351 25930 consensus_queue.cc:237] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.139130 24450 catalog_manager.cc:5484] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
W20240501 19:56:39.269186 26371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:39.269518 26371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:39.272979 26371 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 19:56:39.379115 24595 raft_consensus.cc:1261] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.379509 24725 raft_consensus.cc:1261] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.380105 26366 consensus_queue.cc:1035] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.380767 26366 consensus_queue.cc:1035] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.386478 24856 raft_consensus.cc:1261] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.387006 26013 consensus_queue.cc:1035] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.387300 26408 mvcc.cc:205] Tried to move back new op lower bound from 7022974563852566528 to 7022974562026049536. Current Snapshot: MvccSnapshot[applied={T|T < 7022974563852566528}]
I20240501 19:56:39.388000 24989 raft_consensus.cc:1261] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.388648 26013 consensus_queue.cc:1035] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.394825 24595 raft_consensus.cc:1261] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.395015 24857 raft_consensus.cc:1261] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.395218 25930 consensus_queue.cc:1035] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.396432 25930 consensus_queue.cc:1035] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.406256 26406 mvcc.cc:205] Tried to move back new op lower bound from 7022974563911925760 to 7022974562867744768. Current Snapshot: MvccSnapshot[applied={T|T < 7022974563911925760}]
I20240501 19:56:39.420145 26410 mvcc.cc:205] Tried to move back new op lower bound from 7022974563882606592 to 7022974562665611264. Current Snapshot: MvccSnapshot[applied={T|T < 7022974563882606592}]
I20240501 19:56:39.440016 26404 mvcc.cc:205] Tried to move back new op lower bound from 7022974563882606592 to 7022974562665611264. Current Snapshot: MvccSnapshot[applied={T|T < 7022974563882606592}]
INSERT report
rows total: 2000
time total: 138.813 ms
time per row: 0.0694067 ms
I20240501 19:56:39.455938 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:39.461421 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 00b49c7daa8e49e9b264fdcab76ea976 (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.461421 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 00b49c7daa8e49e9b264fdcab76ea976 (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.461759 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b49c7daa8e49e9b264fdcab76ea976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.461768 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b49c7daa8e49e9b264fdcab76ea976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.461781 24574 tablet_service.cc:1467] Processing CreateTablet for tablet f6dbe23d20b34a348938dcc3571020b7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.461975 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6dbe23d20b34a348938dcc3571020b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.462425 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 09d33aa5ebfd404ca218b069767d216b (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.462687 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d33aa5ebfd404ca218b069767d216b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.463222 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 00b49c7daa8e49e9b264fdcab76ea976 (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.463418 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b49c7daa8e49e9b264fdcab76ea976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.464633 24968 tablet_service.cc:1467] Processing CreateTablet for tablet f6dbe23d20b34a348938dcc3571020b7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.464699 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 09d33aa5ebfd404ca218b069767d216b (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.464910 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6dbe23d20b34a348938dcc3571020b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.465193 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d33aa5ebfd404ca218b069767d216b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.466361 26433 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:39.466810 26435 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:39.467375 26433 tablet_bootstrap.cc:654] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.467371 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 09d33aa5ebfd404ca218b069767d216b (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.467630 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d33aa5ebfd404ca218b069767d216b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.467720 26435 tablet_bootstrap.cc:654] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.467846 26434 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:39.468816 26434 tablet_bootstrap.cc:654] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.469026 26435 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:39.469116 26435 ts_tablet_manager.cc:1367] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:39.469895 26435 raft_consensus.cc:348] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.470036 26435 raft_consensus.cc:374] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.470072 26435 raft_consensus.cc:729] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.470230 26435 consensus_queue.cc:260] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.470317 26433 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:39.470523 26435 ts_tablet_manager.cc:1398] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.470711 26435 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:39.471148 26433 ts_tablet_manager.cc:1367] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:39.471293 26434 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:39.471361 26434 ts_tablet_manager.cc:1367] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:39.471630 26433 raft_consensus.cc:348] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.471724 26433 raft_consensus.cc:374] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.471757 26433 raft_consensus.cc:729] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.471791 26435 tablet_bootstrap.cc:654] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.471758 26434 raft_consensus.cc:348] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.471848 26434 raft_consensus.cc:374] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.471879 26434 raft_consensus.cc:729] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.471880 26433 consensus_queue.cc:260] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.471985 26434 consensus_queue.cc:260] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.472174 26433 ts_tablet_manager.cc:1398] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.472229 26434 ts_tablet_manager.cc:1398] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:39.472347 26433 tablet_bootstrap.cc:492] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:39.472365 26434 tablet_bootstrap.cc:492] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:39.473273 26434 tablet_bootstrap.cc:654] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.473364 26433 tablet_bootstrap.cc:654] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.473908 24706 tablet_service.cc:1467] Processing CreateTablet for tablet f6dbe23d20b34a348938dcc3571020b7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.474208 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6dbe23d20b34a348938dcc3571020b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.475598 26432 tablet_bootstrap.cc:492] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:39.476482 26432 tablet_bootstrap.cc:654] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.477627 26435 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:39.477735 26435 ts_tablet_manager.cc:1367] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:39.477797 26432 tablet_bootstrap.cc:492] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:39.477897 26432 ts_tablet_manager.cc:1367] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:39.478221 26435 raft_consensus.cc:348] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.478348 26435 raft_consensus.cc:374] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.478394 26435 raft_consensus.cc:729] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.478348 26432 raft_consensus.cc:348] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.478461 26432 raft_consensus.cc:374] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.478490 26432 raft_consensus.cc:729] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.478518 26435 consensus_queue.cc:260] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.478624 26369 raft_consensus.cc:482] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.478629 26432 consensus_queue.cc:260] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.478703 26369 raft_consensus.cc:504] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.478912 26432 ts_tablet_manager.cc:1398] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.479074 26435 ts_tablet_manager.cc:1398] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:39.479137 26369 leader_election.cc:288] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:39.479159 26432 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:39.479732 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d33aa5ebfd404ca218b069767d216b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:39.480075 26432 tablet_bootstrap.cc:654] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.480240 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d33aa5ebfd404ca218b069767d216b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
W20240501 19:56:39.480268 24922 leader_election.cc:341] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:39.480367 24857 raft_consensus.cc:2454] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:39.480698 24923 leader_election.cc:302] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 9081292597be4a72a42b216fddcd7971
I20240501 19:56:39.480960 25930 raft_consensus.cc:2781] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.481007 25930 raft_consensus.cc:482] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.481046 25930 raft_consensus.cc:3037] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.481130 26432 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:39.481202 26432 ts_tablet_manager.cc:1367] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:39.481568 26432 raft_consensus.cc:348] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.481653 26432 raft_consensus.cc:374] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.481683 26432 raft_consensus.cc:729] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.481775 26433 tablet_bootstrap.cc:492] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:39.481858 26433 ts_tablet_manager.cc:1367] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:39.481808 26432 consensus_queue.cc:260] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.481992 25930 raft_consensus.cc:504] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.482115 26432 ts_tablet_manager.cc:1398] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:39.482280 26432 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:39.482230 26433 raft_consensus.cc:348] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.482324 26433 raft_consensus.cc:374] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.482321 25930 leader_election.cc:288] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:39.482357 26433 raft_consensus.cc:729] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.482491 26433 consensus_queue.cc:260] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.482775 26433 ts_tablet_manager.cc:1398] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.483011 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d33aa5ebfd404ca218b069767d216b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:39.483156 24725 raft_consensus.cc:3037] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.483199 26432 tablet_bootstrap.cc:654] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.483173 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d33aa5ebfd404ca218b069767d216b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:39.483276 24857 raft_consensus.cc:3037] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.484030 24725 raft_consensus.cc:2454] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:39.484118 26432 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:39.484207 26432 ts_tablet_manager.cc:1367] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:39.484244 24857 raft_consensus.cc:2454] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:39.484347 24922 leader_election.cc:302] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:39.484509 25930 raft_consensus.cc:2781] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.484572 25930 raft_consensus.cc:686] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:39.484647 26432 raft_consensus.cc:348] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.484730 26432 raft_consensus.cc:374] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.484696 25930 consensus_queue.cc:237] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.484758 26432 raft_consensus.cc:729] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.484862 26432 consensus_queue.cc:260] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.485113 26432 ts_tablet_manager.cc:1398] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:39.488175 24445 catalog_manager.cc:5484] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:39.489620 26434 tablet_bootstrap.cc:492] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:39.489722 26434 ts_tablet_manager.cc:1367] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.001s
I20240501 19:56:39.490182 26434 raft_consensus.cc:348] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.490274 26434 raft_consensus.cc:374] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.490314 26434 raft_consensus.cc:729] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.490481 26434 consensus_queue.cc:260] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.490799 26434 ts_tablet_manager.cc:1398] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:39.599575 26145 raft_consensus.cc:482] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.599733 26145 raft_consensus.cc:504] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.600216 26145 leader_election.cc:288] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:39.600592 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6dbe23d20b34a348938dcc3571020b7" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:39.600607 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6dbe23d20b34a348938dcc3571020b7" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:39.600733 24725 raft_consensus.cc:2454] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:39.600757 24989 raft_consensus.cc:2454] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:39.601015 24529 leader_election.cc:302] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:39.601224 26394 raft_consensus.cc:2781] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.601282 26394 raft_consensus.cc:482] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.601330 26394 raft_consensus.cc:3037] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.602309 26394 raft_consensus.cc:504] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.602622 26394 leader_election.cc:288] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:39.602912 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6dbe23d20b34a348938dcc3571020b7" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:39.602936 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6dbe23d20b34a348938dcc3571020b7" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:39.603055 24989 raft_consensus.cc:3037] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.603055 24725 raft_consensus.cc:3037] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.603960 24989 raft_consensus.cc:2454] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:39.603960 24725 raft_consensus.cc:2454] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:39.604243 24530 leader_election.cc:302] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:39.604393 26394 raft_consensus.cc:2781] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.604490 26394 raft_consensus.cc:686] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:39.604643 26394 consensus_queue.cc:237] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:39.606359 24445 catalog_manager.cc:5484] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:39.613451 26427 raft_consensus.cc:482] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.613548 26427 raft_consensus.cc:504] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.613900 26427 leader_election.cc:288] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:39.614174 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b49c7daa8e49e9b264fdcab76ea976" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:39.614291 24594 raft_consensus.cc:2454] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:39.614269 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b49c7daa8e49e9b264fdcab76ea976" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:39.614403 24857 raft_consensus.cc:2454] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:39.614578 24660 leader_election.cc:302] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:39.614784 26427 raft_consensus.cc:2781] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.614842 26427 raft_consensus.cc:482] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.614904 26427 raft_consensus.cc:3037] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.615816 26427 raft_consensus.cc:504] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.616114 26427 leader_election.cc:288] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:39.616361 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b49c7daa8e49e9b264fdcab76ea976" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:39.616411 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b49c7daa8e49e9b264fdcab76ea976" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:39.616487 24594 raft_consensus.cc:3037] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.616503 24857 raft_consensus.cc:3037] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.617202 24857 raft_consensus.cc:2454] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:39.617264 24594 raft_consensus.cc:2454] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:39.617502 24661 leader_election.cc:302] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:39.617662 26427 raft_consensus.cc:2781] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.617746 26427 raft_consensus.cc:686] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:39.617877 26427 consensus_queue.cc:237] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.619588 24445 catalog_manager.cc:5484] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
W20240501 19:56:39.743399 26448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:39.743707 26448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:39.747833 26448 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 19:56:39.844036 24989 raft_consensus.cc:1261] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.844377 24725 raft_consensus.cc:1261] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.844719 26394 consensus_queue.cc:1035] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.845067 26394 consensus_queue.cc:1035] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.847306 24857 raft_consensus.cc:1261] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.847515 24727 raft_consensus.cc:1261] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.847720 26013 consensus_queue.cc:1035] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.848714 25930 consensus_queue.cc:1035] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.854355 24856 raft_consensus.cc:1261] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.855000 26369 consensus_queue.cc:1035] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.855849 24594 raft_consensus.cc:1261] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.857720 26291 consensus_queue.cc:1035] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.159 ms
time per row: 0.0570794 ms
I20240501 19:56:39.911639 24447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:39.917083 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 23ae3af247c1456e8b75640c25906faf (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.917411 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ae3af247c1456e8b75640c25906faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.917559 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 23ae3af247c1456e8b75640c25906faf (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.917611 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 0064e3e3ae1548bba511bd76fb4ec3e1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.917819 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ae3af247c1456e8b75640c25906faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.917866 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0064e3e3ae1548bba511bd76fb4ec3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.918401 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed4b95e555a46c5976ca9e084737ea1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.918673 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed4b95e555a46c5976ca9e084737ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.919852 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed4b95e555a46c5976ca9e084737ea1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.920047 26433 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:39.920097 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed4b95e555a46c5976ca9e084737ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.920073 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 0064e3e3ae1548bba511bd76fb4ec3e1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.920221 26435 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:39.920259 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0064e3e3ae1548bba511bd76fb4ec3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.921067 26435 tablet_bootstrap.cc:654] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.921010 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 0064e3e3ae1548bba511bd76fb4ec3e1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.921308 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0064e3e3ae1548bba511bd76fb4ec3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.922230 26433 tablet_bootstrap.cc:654] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.922412 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 23ae3af247c1456e8b75640c25906faf (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.922653 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ae3af247c1456e8b75640c25906faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.922859 26432 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:39.923547 26434 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:39.923717 26432 tablet_bootstrap.cc:654] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.924371 26434 tablet_bootstrap.cc:654] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.924465 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed4b95e555a46c5976ca9e084737ea1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.924662 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed4b95e555a46c5976ca9e084737ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.925225 26435 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:39.925323 26435 ts_tablet_manager.cc:1367] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:39.925815 26432 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:39.925803 26435 raft_consensus.cc:348] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.925884 26432 ts_tablet_manager.cc:1367] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:39.925911 26435 raft_consensus.cc:374] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.925941 26435 raft_consensus.cc:729] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.926059 26435 consensus_queue.cc:260] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.926328 26435 ts_tablet_manager.cc:1398] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.926277 26432 raft_consensus.cc:348] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.926368 26432 raft_consensus.cc:374] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.926398 26432 raft_consensus.cc:729] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.926540 26435 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:39.926550 26432 consensus_queue.cc:260] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.926818 26432 ts_tablet_manager.cc:1398] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.927409 26435 tablet_bootstrap.cc:654] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.928781 26435 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:39.928858 26435 ts_tablet_manager.cc:1367] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:39.929260 26435 raft_consensus.cc:348] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.929348 26435 raft_consensus.cc:374] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.929380 26435 raft_consensus.cc:729] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.929502 26435 consensus_queue.cc:260] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.929781 26435 ts_tablet_manager.cc:1398] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:39.930526 26433 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:39.930616 26433 ts_tablet_manager.cc:1367] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:56:39.931097 26433 raft_consensus.cc:348] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.931218 26433 raft_consensus.cc:374] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.931279 26433 raft_consensus.cc:729] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.931408 26433 consensus_queue.cc:260] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.931691 26433 ts_tablet_manager.cc:1398] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.931887 26433 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:39.932006 26434 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:39.932086 26434 ts_tablet_manager.cc:1367] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 19:56:39.932540 26434 raft_consensus.cc:348] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.932653 26434 raft_consensus.cc:374] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.932694 26434 raft_consensus.cc:729] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.932817 26433 tablet_bootstrap.cc:654] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.932822 26434 consensus_queue.cc:260] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.933108 26434 ts_tablet_manager.cc:1398] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:39.933300 26434 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:39.934039 26434 tablet_bootstrap.cc:654] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.935671 26434 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:39.935740 26434 ts_tablet_manager.cc:1367] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:39.936097 26434 raft_consensus.cc:348] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.936187 26434 raft_consensus.cc:374] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.936213 26434 raft_consensus.cc:729] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.936344 26434 consensus_queue.cc:260] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:39.936666 26434 ts_tablet_manager.cc:1398] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:39.936805 26434 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:39.937589 26434 tablet_bootstrap.cc:654] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.938412 26433 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:39.938521 26433 ts_tablet_manager.cc:1367] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:39.939265 26433 raft_consensus.cc:348] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.939379 26433 raft_consensus.cc:374] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.939421 26433 raft_consensus.cc:729] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.939587 26433 consensus_queue.cc:260] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:39.939831 26434 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:39.939908 26434 ts_tablet_manager.cc:1367] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:39.939954 26433 ts_tablet_manager.cc:1398] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.940176 26433 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:39.940315 26434 raft_consensus.cc:348] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.940405 26434 raft_consensus.cc:374] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.940435 26434 raft_consensus.cc:729] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.940578 26434 consensus_queue.cc:260] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.940855 26434 ts_tablet_manager.cc:1398] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:39.941711 26433 tablet_bootstrap.cc:654] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.943850 26433 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:39.943915 26433 ts_tablet_manager.cc:1367] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:39.944336 26433 raft_consensus.cc:348] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.944442 26433 raft_consensus.cc:374] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.944465 26433 raft_consensus.cc:729] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.944573 26433 consensus_queue.cc:260] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:39.944833 26433 ts_tablet_manager.cc:1398] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.012761 26485 raft_consensus.cc:482] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.012899 26485 raft_consensus.cc:504] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.013310 26485 leader_election.cc:288] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:40.013701 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0064e3e3ae1548bba511bd76fb4ec3e1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:40.013700 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0064e3e3ae1548bba511bd76fb4ec3e1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:40.013840 24989 raft_consensus.cc:2454] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:40.013840 24858 raft_consensus.cc:2454] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:40.014102 24530 leader_election.cc:302] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.014274 26485 raft_consensus.cc:2781] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.014333 26485 raft_consensus.cc:482] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.014394 26485 raft_consensus.cc:3037] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.015209 26485 raft_consensus.cc:504] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.015470 26485 leader_election.cc:288] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:40.015764 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0064e3e3ae1548bba511bd76fb4ec3e1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:40.015851 24858 raft_consensus.cc:3037] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.015823 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0064e3e3ae1548bba511bd76fb4ec3e1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:40.015955 24989 raft_consensus.cc:3037] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.016603 24858 raft_consensus.cc:2454] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:40.016678 24989 raft_consensus.cc:2454] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:40.016863 24530 leader_election.cc:302] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.016999 26485 raft_consensus.cc:2781] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.017095 26485 raft_consensus.cc:686] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:40.017262 26485 consensus_queue.cc:237] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.018934 24449 catalog_manager.cc:5484] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:40.135686 26485 raft_consensus.cc:482] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.135813 26485 raft_consensus.cc:504] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:40.136188 26485 leader_election.cc:288] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:40.136533 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed4b95e555a46c5976ca9e084737ea1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:40.136598 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed4b95e555a46c5976ca9e084737ea1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:40.136672 24727 raft_consensus.cc:2454] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:40.136716 24858 raft_consensus.cc:2454] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:40.136979 24530 leader_election.cc:302] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.137132 26485 raft_consensus.cc:2781] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.137193 26485 raft_consensus.cc:482] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.137233 26485 raft_consensus.cc:3037] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.138039 26485 raft_consensus.cc:504] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:40.138357 26485 leader_election.cc:288] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:40.138656 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed4b95e555a46c5976ca9e084737ea1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:40.138672 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed4b95e555a46c5976ca9e084737ea1" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:40.138761 24858 raft_consensus.cc:3037] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.138783 24727 raft_consensus.cc:3037] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.139583 24727 raft_consensus.cc:2454] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:40.139689 24858 raft_consensus.cc:2454] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:40.139864 24529 leader_election.cc:302] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.140077 26485 raft_consensus.cc:2781] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.140235 26485 raft_consensus.cc:686] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:40.140388 26485 consensus_queue.cc:237] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:40.142254 24449 catalog_manager.cc:5484] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:40.261178 26369 raft_consensus.cc:482] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.261323 26369 raft_consensus.cc:504] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.261724 26369 leader_election.cc:288] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:40.262132 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ae3af247c1456e8b75640c25906faf" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:40.262161 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ae3af247c1456e8b75640c25906faf" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:40.262290 24594 raft_consensus.cc:2454] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:40.262290 24858 raft_consensus.cc:2454] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:40.262686 24922 leader_election.cc:302] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.262892 26369 raft_consensus.cc:2781] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.262959 26369 raft_consensus.cc:482] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.263005 26369 raft_consensus.cc:3037] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.263865 26369 raft_consensus.cc:504] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.264195 26369 leader_election.cc:288] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:40.264508 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ae3af247c1456e8b75640c25906faf" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:40.264514 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ae3af247c1456e8b75640c25906faf" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:40.264629 24857 raft_consensus.cc:3037] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.264638 24594 raft_consensus.cc:3037] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.265411 24594 raft_consensus.cc:2454] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:40.265414 24857 raft_consensus.cc:2454] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:40.265731 24923 leader_election.cc:302] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:40.265897 26369 raft_consensus.cc:2781] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.265972 26369 raft_consensus.cc:686] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:40.266131 26369 consensus_queue.cc:237] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.268091 24449 catalog_manager.cc:5484] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
W20240501 19:56:40.408785 26499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:40.409035 26499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:40.412209 26499 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:40.424090 26485 consensus_queue.cc:1035] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.427374 26394 consensus_queue.cc:1035] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
I20240501 19:56:40.492139 24594 raft_consensus.cc:1261] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.492139 24858 raft_consensus.cc:1261] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.492533 26369 consensus_queue.cc:1035] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.492846 26369 consensus_queue.cc:1035] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.493054 24727 raft_consensus.cc:1261] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.498005 26394 consensus_queue.cc:1035] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.502067 24858 raft_consensus.cc:1261] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.503260 26394 consensus_queue.cc:1035] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 97.8371 ms
time per row: 0.0489186 ms
I20240501 19:56:40.555222 24449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:40.561309 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1c61cca30c4d878d921c15158a5947 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.561462 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1c61cca30c4d878d921c15158a5947 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.561636 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1c61cca30c4d878d921c15158a5947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.561805 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1c61cca30c4d878d921c15158a5947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.562302 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1c61cca30c4d878d921c15158a5947 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.562563 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1c61cca30c4d878d921c15158a5947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.563762 26556 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:40.564090 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.564373 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.564934 26556 tablet_bootstrap.cc:654] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.564925 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.565140 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.566365 26560 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:40.567318 26556 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:40.567329 26560 tablet_bootstrap.cc:654] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.567409 26556 ts_tablet_manager.cc:1367] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:40.566955 24969 tablet_service.cc:1467] Processing CreateTablet for tablet fe354fb491da4c8d896e46b5e1f6edf2 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.567772 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe354fb491da4c8d896e46b5e1f6edf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.567963 26556 raft_consensus.cc:348] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.568044 26556 raft_consensus.cc:374] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.568064 26556 raft_consensus.cc:729] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.568173 26556 consensus_queue.cc:260] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.568904 26556 ts_tablet_manager.cc:1398] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:40.569747 26558 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:40.570397 24707 tablet_service.cc:1467] Processing CreateTablet for tablet fe354fb491da4c8d896e46b5e1f6edf2 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.570621 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe354fb491da4c8d896e46b5e1f6edf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.570670 26558 tablet_bootstrap.cc:654] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.570664 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.570664 24575 tablet_service.cc:1467] Processing CreateTablet for tablet fe354fb491da4c8d896e46b5e1f6edf2 (DEFAULT_TABLE table=rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.570847 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.570921 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe354fb491da4c8d896e46b5e1f6edf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.573450 26556 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:40.574182 26556 tablet_bootstrap.cc:654] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.574386 26557 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:40.575387 26557 tablet_bootstrap.cc:654] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.575645 26556 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:40.575755 26556 ts_tablet_manager.cc:1367] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:40.576123 26556 raft_consensus.cc:348] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.576215 26556 raft_consensus.cc:374] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.576249 26556 raft_consensus.cc:729] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.576412 26556 consensus_queue.cc:260] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.577124 26560 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:40.577205 26560 ts_tablet_manager.cc:1367] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:40.577586 26560 raft_consensus.cc:348] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.577706 26560 raft_consensus.cc:374] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.577733 26560 raft_consensus.cc:729] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.577785 26556 ts_tablet_manager.cc:1398] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:40.577827 26560 consensus_queue.cc:260] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.577942 26556 tablet_bootstrap.cc:492] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:40.578063 26560 ts_tablet_manager.cc:1398] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.578773 26556 tablet_bootstrap.cc:654] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.579514 26557 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:40.579588 26557 ts_tablet_manager.cc:1367] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:40.579967 26556 tablet_bootstrap.cc:492] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:40.580042 26556 ts_tablet_manager.cc:1367] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:40.580324 26557 raft_consensus.cc:348] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.580422 26557 raft_consensus.cc:374] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.580449 26557 raft_consensus.cc:729] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.580439 26556 raft_consensus.cc:348] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.580530 26556 raft_consensus.cc:374] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.580561 26556 raft_consensus.cc:729] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.580579 26557 consensus_queue.cc:260] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.580675 26556 consensus_queue.cc:260] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.580938 26556 ts_tablet_manager.cc:1398] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:40.581066 26557 ts_tablet_manager.cc:1398] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.581403 26557 tablet_bootstrap.cc:492] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:40.582324 26557 tablet_bootstrap.cc:654] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.583622 26557 tablet_bootstrap.cc:492] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:40.583654 26558 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:40.583712 26557 ts_tablet_manager.cc:1367] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:40.583738 26558 ts_tablet_manager.cc:1367] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:40.584110 26557 raft_consensus.cc:348] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.584213 26557 raft_consensus.cc:374] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.584158 26558 raft_consensus.cc:348] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.584241 26557 raft_consensus.cc:729] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.584251 26558 raft_consensus.cc:374] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.584302 26558 raft_consensus.cc:729] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.584367 26557 consensus_queue.cc:260] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.584426 26558 consensus_queue.cc:260] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.584653 26557 ts_tablet_manager.cc:1398] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.585322 26558 ts_tablet_manager.cc:1398] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:40.585513 26558 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:40.586396 26558 tablet_bootstrap.cc:654] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.588214 26558 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:40.588299 26558 ts_tablet_manager.cc:1367] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:40.588660 26558 raft_consensus.cc:348] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.588784 26558 raft_consensus.cc:374] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.588821 26558 raft_consensus.cc:729] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.588930 26558 consensus_queue.cc:260] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.589195 26558 ts_tablet_manager.cc:1398] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.589397 26558 tablet_bootstrap.cc:492] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:40.590343 26558 tablet_bootstrap.cc:654] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.591564 26558 tablet_bootstrap.cc:492] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:40.591699 26558 ts_tablet_manager.cc:1367] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:40.592133 26558 raft_consensus.cc:348] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.592408 26558 raft_consensus.cc:374] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.592463 26558 raft_consensus.cc:729] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.592569 26558 consensus_queue.cc:260] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.592922 26558 ts_tablet_manager.cc:1398] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.680392 26291 raft_consensus.cc:482] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.680527 26291 raft_consensus.cc:504] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.680930 26291 leader_election.cc:288] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:40.681308 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354fb491da4c8d896e46b5e1f6edf2" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:40.681456 24594 raft_consensus.cc:2454] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:40.681543 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354fb491da4c8d896e46b5e1f6edf2" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:40.681723 24986 raft_consensus.cc:2454] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:40.681761 24660 leader_election.cc:302] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.681926 26291 raft_consensus.cc:2781] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.681989 26291 raft_consensus.cc:482] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.682041 26291 raft_consensus.cc:3037] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.682880 26291 raft_consensus.cc:504] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.683256 26291 leader_election.cc:288] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:40.683540 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354fb491da4c8d896e46b5e1f6edf2" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:40.683569 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe354fb491da4c8d896e46b5e1f6edf2" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:40.683646 24594 raft_consensus.cc:3037] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.683686 24986 raft_consensus.cc:3037] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.684414 24594 raft_consensus.cc:2454] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:40.684473 24986 raft_consensus.cc:2454] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:40.684688 24660 leader_election.cc:302] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.684878 26291 raft_consensus.cc:2781] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.685031 26291 raft_consensus.cc:686] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:40.685159 26291 consensus_queue.cc:237] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:40.687158 24449 catalog_manager.cc:5484] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:40.729799 26486 raft_consensus.cc:482] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.729936 26486 raft_consensus.cc:504] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.730383 26486 leader_election.cc:288] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:40.730751 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2d49dcdd8f49d48d8a9bd6d10c332c" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:40.730801 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2d49dcdd8f49d48d8a9bd6d10c332c" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:40.730885 24727 raft_consensus.cc:2454] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:40.730935 24986 raft_consensus.cc:2454] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:40.731246 24791 leader_election.cc:302] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:40.731420 26486 raft_consensus.cc:2781] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.731479 26486 raft_consensus.cc:482] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.731505 26486 raft_consensus.cc:3037] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.732344 26486 raft_consensus.cc:504] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.732668 26486 leader_election.cc:288] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:40.732980 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2d49dcdd8f49d48d8a9bd6d10c332c" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:40.732991 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2d49dcdd8f49d48d8a9bd6d10c332c" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:40.733091 24727 raft_consensus.cc:3037] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.733079 24986 raft_consensus.cc:3037] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.733844 24986 raft_consensus.cc:2454] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:40.733850 24727 raft_consensus.cc:2454] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:40.734091 24791 leader_election.cc:302] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:40.734247 26486 raft_consensus.cc:2781] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.734328 26486 raft_consensus.cc:686] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:40.734465 26486 consensus_queue.cc:237] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:40.736413 24449 catalog_manager.cc:5484] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:40.812155 26394 raft_consensus.cc:482] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.812296 26394 raft_consensus.cc:504] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.812776 26394 leader_election.cc:288] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:40.813035 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1c61cca30c4d878d921c15158a5947" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:40.813179 24727 raft_consensus.cc:2454] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:40.813216 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1c61cca30c4d878d921c15158a5947" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:40.813359 24986 raft_consensus.cc:2454] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:40.813458 24529 leader_election.cc:302] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.813624 26394 raft_consensus.cc:2781] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.813676 26394 raft_consensus.cc:482] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.813721 26394 raft_consensus.cc:3037] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.814589 26394 raft_consensus.cc:504] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.814908 26394 leader_election.cc:288] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:40.815244 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1c61cca30c4d878d921c15158a5947" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:40.815295 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1c61cca30c4d878d921c15158a5947" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:40.815366 24727 raft_consensus.cc:3037] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.815397 24986 raft_consensus.cc:3037] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.816124 24727 raft_consensus.cc:2454] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:40.816265 24986 raft_consensus.cc:2454] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:40.816447 24529 leader_election.cc:302] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:40.816597 26394 raft_consensus.cc:2781] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.816704 26394 raft_consensus.cc:686] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:40.816844 26394 consensus_queue.cc:237] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:40.818614 24449 catalog_manager.cc:5484] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
W20240501 19:56:40.980206 26573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:40.980443 26573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:40.983498 26573 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 19:56:41.062826 24986 raft_consensus.cc:1261] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.063045 24727 raft_consensus.cc:1261] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.063292 26394 consensus_queue.cc:1035] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.064522 26549 consensus_queue.cc:1035] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.086947 24986 raft_consensus.cc:1261] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.087163 24594 raft_consensus.cc:1261] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.087340 26291 consensus_queue.cc:1035] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.087575 26569 consensus_queue.cc:1035] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.092289 24727 raft_consensus.cc:1261] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.092665 26366 consensus_queue.cc:1035] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.093008 24989 raft_consensus.cc:1261] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.093487 26366 consensus_queue.cc:1035] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.543 ms
time per row: 0.0592713 ms
I20240501 19:56:41.144117 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:41.149804 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 224ae792734e4d1ab3c28e20d82fdf14 (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.150111 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224ae792734e4d1ab3c28e20d82fdf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.150718 24838 tablet_service.cc:1467] Processing CreateTablet for tablet 718c7f2f1f074e3c84534b502a4809a2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.150765 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 224ae792734e4d1ab3c28e20d82fdf14 (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.150796 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 164460e6fbc44fb1a9a12d52fea9254b (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.151000 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718c7f2f1f074e3c84534b502a4809a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.151083 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164460e6fbc44fb1a9a12d52fea9254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.151217 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 164460e6fbc44fb1a9a12d52fea9254b (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.151430 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164460e6fbc44fb1a9a12d52fea9254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.153090 26632 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:41.153261 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 718c7f2f1f074e3c84534b502a4809a2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.153488 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718c7f2f1f074e3c84534b502a4809a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.153861 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 718c7f2f1f074e3c84534b502a4809a2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.154093 26632 tablet_bootstrap.cc:654] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.154105 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718c7f2f1f074e3c84534b502a4809a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.154147 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 224ae792734e4d1ab3c28e20d82fdf14 (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.154388 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224ae792734e4d1ab3c28e20d82fdf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.155246 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 164460e6fbc44fb1a9a12d52fea9254b (DEFAULT_TABLE table=rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.155493 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164460e6fbc44fb1a9a12d52fea9254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.156667 26635 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:41.157508 26635 tablet_bootstrap.cc:654] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.157575 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224ae792734e4d1ab3c28e20d82fdf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.158654 26631 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:41.160832 26633 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:41.160876 26631 tablet_bootstrap.cc:654] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.160952 26632 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:41.161041 26632 ts_tablet_manager.cc:1367] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:41.161432 26632 raft_consensus.cc:348] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.161556 26632 raft_consensus.cc:374] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.161600 26632 raft_consensus.cc:729] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.161753 26632 consensus_queue.cc:260] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.161851 26633 tablet_bootstrap.cc:654] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.162019 26632 ts_tablet_manager.cc:1398] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.162333 26631 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:41.162431 26631 ts_tablet_manager.cc:1367] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:41.162580 26635 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:41.162806 26632 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:41.162962 26633 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:41.163007 26635 ts_tablet_manager.cc:1367] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:41.163060 26633 ts_tablet_manager.cc:1367] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:41.163498 26633 raft_consensus.cc:348] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.163596 26633 raft_consensus.cc:374] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.163533 26635 raft_consensus.cc:348] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.163630 26633 raft_consensus.cc:729] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.163646 26635 raft_consensus.cc:374] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.163690 26635 raft_consensus.cc:729] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.163775 26632 tablet_bootstrap.cc:654] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.163751 26633 consensus_queue.cc:260] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.163861 26635 consensus_queue.cc:260] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.164026 26633 ts_tablet_manager.cc:1398] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.164201 26633 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:41.164777 26635 ts_tablet_manager.cc:1398] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:41.164935 26635 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:41.165347 26633 tablet_bootstrap.cc:654] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.165874 26635 tablet_bootstrap.cc:654] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.169227 26635 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:41.169317 26635 ts_tablet_manager.cc:1367] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:41.169675 26635 raft_consensus.cc:348] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.169760 26635 raft_consensus.cc:374] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.169790 26635 raft_consensus.cc:729] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.169883 26635 consensus_queue.cc:260] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.170135 26635 ts_tablet_manager.cc:1398] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.170678 26631 raft_consensus.cc:348] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.170787 26631 raft_consensus.cc:374] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.170821 26631 raft_consensus.cc:729] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.170970 26631 consensus_queue.cc:260] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.171710 26633 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:41.171803 26633 ts_tablet_manager.cc:1367] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:41.172025 26632 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:41.172099 26632 ts_tablet_manager.cc:1367] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:41.172238 26633 raft_consensus.cc:348] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.172340 26633 raft_consensus.cc:374] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.172370 26633 raft_consensus.cc:729] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.172478 26633 consensus_queue.cc:260] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.172516 26632 raft_consensus.cc:348] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.172617 26632 raft_consensus.cc:374] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.172645 26632 raft_consensus.cc:729] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.172750 26632 consensus_queue.cc:260] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.172770 26633 ts_tablet_manager.cc:1398] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.172984 26632 ts_tablet_manager.cc:1398] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.173128 26632 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:41.173687 26631 ts_tablet_manager.cc:1398] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:41.174041 26632 tablet_bootstrap.cc:654] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.175159 26631 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:41.176184 26631 tablet_bootstrap.cc:654] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.176771 26632 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:41.176853 26632 ts_tablet_manager.cc:1367] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:41.177296 26631 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:41.177299 26632 raft_consensus.cc:348] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.177388 26631 ts_tablet_manager.cc:1367] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:41.177469 26632 raft_consensus.cc:374] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.177523 26632 raft_consensus.cc:729] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.177649 26632 consensus_queue.cc:260] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.177840 26631 raft_consensus.cc:348] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.177940 26631 raft_consensus.cc:374] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.177973 26632 ts_tablet_manager.cc:1398] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.177973 26631 raft_consensus.cc:729] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.178090 26631 consensus_queue.cc:260] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.178359 26631 ts_tablet_manager.cc:1398] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.251952 26569 raft_consensus.cc:482] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.252084 26569 raft_consensus.cc:504] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.252445 26569 leader_election.cc:288] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:41.252830 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:41.252843 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:41.252959 24856 raft_consensus.cc:2454] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:41.252980 24594 raft_consensus.cc:2454] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:41.253280 24661 leader_election.cc:302] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:41.253484 26569 raft_consensus.cc:2781] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.253530 26569 raft_consensus.cc:482] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.253571 26569 raft_consensus.cc:3037] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.254612 26569 raft_consensus.cc:504] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.254952 26569 leader_election.cc:288] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:41.255388 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:41.255409 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:41.255512 24856 raft_consensus.cc:3037] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.255523 24594 raft_consensus.cc:3037] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.256469 24594 raft_consensus.cc:2454] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:41.256469 24856 raft_consensus.cc:2454] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:41.256785 24660 leader_election.cc:302] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:41.256951 26569 raft_consensus.cc:2781] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.257059 26569 raft_consensus.cc:686] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:41.257195 26569 consensus_queue.cc:237] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.258981 24451 catalog_manager.cc:5484] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:41.263995 26615 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.264109 26615 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.264448 26615 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:41.264825 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:41.264838 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:41.264951 24856 raft_consensus.cc:2454] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:41.264961 24986 raft_consensus.cc:2454] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:41.265245 24530 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:41.265420 26615 raft_consensus.cc:2781] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.265481 26615 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.265523 26615 raft_consensus.cc:3037] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.266285 26615 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.266644 26615 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:41.266906 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:41.266983 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:41.267017 24986 raft_consensus.cc:3037] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.267117 24856 raft_consensus.cc:3037] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.267872 24986 raft_consensus.cc:2454] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:41.267899 24856 raft_consensus.cc:2454] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:41.268139 24530 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:41.268388 26615 raft_consensus.cc:2781] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.268469 26615 raft_consensus.cc:686] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:41.268600 26615 consensus_queue.cc:237] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:41.270615 24451 catalog_manager.cc:5484] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:41.279486 26569 raft_consensus.cc:482] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.279623 26569 raft_consensus.cc:504] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.280025 26569 leader_election.cc:288] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:41.280458 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164460e6fbc44fb1a9a12d52fea9254b" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:41.280531 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164460e6fbc44fb1a9a12d52fea9254b" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:41.280587 24856 raft_consensus.cc:2454] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:41.280689 24986 raft_consensus.cc:2454] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:41.280869 24661 leader_election.cc:302] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:41.281037 26569 raft_consensus.cc:2781] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.281096 26569 raft_consensus.cc:482] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.281148 26569 raft_consensus.cc:3037] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.281916 26569 raft_consensus.cc:504] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.282225 26569 leader_election.cc:288] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:41.282493 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164460e6fbc44fb1a9a12d52fea9254b" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:41.282562 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164460e6fbc44fb1a9a12d52fea9254b" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:41.282596 24856 raft_consensus.cc:3037] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.282670 24986 raft_consensus.cc:3037] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.283404 24856 raft_consensus.cc:2454] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:41.283473 24986 raft_consensus.cc:2454] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:41.283668 24661 leader_election.cc:302] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:41.283808 26569 raft_consensus.cc:2781] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.283921 26569 raft_consensus.cc:686] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:41.284106 26569 consensus_queue.cc:237] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.285984 24451 catalog_manager.cc:5484] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
W20240501 19:56:41.423563 26647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:41.423908 26647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:41.427505 26647 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240501 19:56:41.517510 24986 raft_consensus.cc:1261] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.518420 24594 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.518733 24857 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.518920 26569 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.519156 24856 raft_consensus.cc:1261] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.520536 26291 consensus_queue.cc:1035] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.520942 26291 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.521212 26291 consensus_queue.cc:1035] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.527855 24989 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.528400 26615 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.528527 24858 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.529325 26615 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.117 ms
time per row: 0.0510587 ms
I20240501 19:56:41.575781 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:41.582325 24837 tablet_service.cc:1467] Processing CreateTablet for tablet a40a737a78824728b9e85e6a4ded638a (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.582635 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a737a78824728b9e85e6a4ded638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.583418 24707 tablet_service.cc:1467] Processing CreateTablet for tablet d56fe9e75a9c418f817185542ae4c1d9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.583674 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56fe9e75a9c418f817185542ae4c1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.585037 24706 tablet_service.cc:1467] Processing CreateTablet for tablet a40a737a78824728b9e85e6a4ded638a (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.585230 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a737a78824728b9e85e6a4ded638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.585796 26635 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:41.586694 26635 tablet_bootstrap.cc:654] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.587204 26632 tablet_bootstrap.cc:492] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:41.588109 26632 tablet_bootstrap.cc:654] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.589258 24575 tablet_service.cc:1467] Processing CreateTablet for tablet a40a737a78824728b9e85e6a4ded638a (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.589258 24576 tablet_service.cc:1467] Processing CreateTablet for tablet e1141ea148d74ed594effc258ad44dd1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.589515 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a737a78824728b9e85e6a4ded638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.589655 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1141ea148d74ed594effc258ad44dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.589887 24574 tablet_service.cc:1467] Processing CreateTablet for tablet d56fe9e75a9c418f817185542ae4c1d9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.590086 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56fe9e75a9c418f817185542ae4c1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.591666 26631 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:41.592468 26631 tablet_bootstrap.cc:654] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.593349 26631 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:41.593428 26631 ts_tablet_manager.cc:1367] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:41.593839 26631 raft_consensus.cc:348] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.593950 26631 raft_consensus.cc:374] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.593983 26631 raft_consensus.cc:729] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.594105 26631 consensus_queue.cc:260] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.594374 26631 ts_tablet_manager.cc:1398] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.594339 24969 tablet_service.cc:1467] Processing CreateTablet for tablet d56fe9e75a9c418f817185542ae4c1d9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.594390 24967 tablet_service.cc:1467] Processing CreateTablet for tablet e1141ea148d74ed594effc258ad44dd1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.594538 26631 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:41.594578 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56fe9e75a9c418f817185542ae4c1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.594713 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1141ea148d74ed594effc258ad44dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.595372 26631 tablet_bootstrap.cc:654] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.596709 26633 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:41.597637 26633 tablet_bootstrap.cc:654] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.598615 24837 tablet_service.cc:1467] Processing CreateTablet for tablet e1141ea148d74ed594effc258ad44dd1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.598819 26631 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:41.598866 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1141ea148d74ed594effc258ad44dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.598963 26631 ts_tablet_manager.cc:1367] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:41.599493 26631 raft_consensus.cc:348] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.599637 26631 raft_consensus.cc:374] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.599706 26631 raft_consensus.cc:729] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.599900 26631 consensus_queue.cc:260] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.600258 26631 ts_tablet_manager.cc:1398] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.600456 26631 tablet_bootstrap.cc:492] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:41.601291 26631 tablet_bootstrap.cc:654] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.603605 26631 tablet_bootstrap.cc:492] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:41.603678 26631 ts_tablet_manager.cc:1367] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:41.604132 26631 raft_consensus.cc:348] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.604212 26631 raft_consensus.cc:374] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.604243 26631 raft_consensus.cc:729] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.604344 26631 consensus_queue.cc:260] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.604620 26631 ts_tablet_manager.cc:1398] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.611662 26632 tablet_bootstrap.cc:492] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:41.611757 26632 ts_tablet_manager.cc:1367] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 19:56:41.611877 26635 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:41.611979 26635 ts_tablet_manager.cc:1367] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
I20240501 19:56:41.612154 26632 raft_consensus.cc:348] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.612250 26632 raft_consensus.cc:374] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.612280 26632 raft_consensus.cc:729] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.612403 26632 consensus_queue.cc:260] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.612408 26635 raft_consensus.cc:348] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.612496 26635 raft_consensus.cc:374] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.612526 26635 raft_consensus.cc:729] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.612632 26635 consensus_queue.cc:260] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.612691 26632 ts_tablet_manager.cc:1398] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.612887 26635 ts_tablet_manager.cc:1398] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.612880 26632 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:41.613070 26635 tablet_bootstrap.cc:492] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:41.613484 26633 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:41.613570 26633 ts_tablet_manager.cc:1367] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240501 19:56:41.613963 26635 tablet_bootstrap.cc:654] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.614007 26632 tablet_bootstrap.cc:654] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.614004 26633 raft_consensus.cc:348] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.614105 26633 raft_consensus.cc:374] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.614142 26633 raft_consensus.cc:729] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.614253 26633 consensus_queue.cc:260] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.614545 26633 ts_tablet_manager.cc:1398] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.614730 26633 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:41.615569 26633 tablet_bootstrap.cc:654] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.615815 26635 tablet_bootstrap.cc:492] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:41.615895 26635 ts_tablet_manager.cc:1367] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:41.616648 26632 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:41.616725 26632 ts_tablet_manager.cc:1367] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:41.617017 26635 raft_consensus.cc:348] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.617146 26633 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:41.617134 26632 raft_consensus.cc:348] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.617216 26632 raft_consensus.cc:374] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.617225 26633 ts_tablet_manager.cc:1367] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:41.617245 26632 raft_consensus.cc:729] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.617108 26635 raft_consensus.cc:374] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.617417 26635 raft_consensus.cc:729] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.617377 26632 consensus_queue.cc:260] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.617563 26635 consensus_queue.cc:260] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.617678 26632 ts_tablet_manager.cc:1398] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.617722 26633 raft_consensus.cc:348] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.617815 26633 raft_consensus.cc:374] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.617820 26635 ts_tablet_manager.cc:1398] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:41.617851 26633 raft_consensus.cc:729] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.617967 26633 consensus_queue.cc:260] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.618345 26633 ts_tablet_manager.cc:1398] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.709060 26366 raft_consensus.cc:482] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.709167 26366 raft_consensus.cc:504] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.709511 26366 leader_election.cc:288] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:41.709834 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1141ea148d74ed594effc258ad44dd1" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:41.709900 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1141ea148d74ed594effc258ad44dd1" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:41.709954 24594 raft_consensus.cc:2454] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:41.710041 24989 raft_consensus.cc:2454] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:41.710250 24791 leader_election.cc:302] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:41.710417 26366 raft_consensus.cc:2781] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.710470 26366 raft_consensus.cc:482] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.710497 26366 raft_consensus.cc:3037] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.711426 26366 raft_consensus.cc:504] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.711719 26366 leader_election.cc:288] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:41.712025 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1141ea148d74ed594effc258ad44dd1" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:41.712016 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1141ea148d74ed594effc258ad44dd1" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:41.712144 24989 raft_consensus.cc:3037] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.712145 24594 raft_consensus.cc:3037] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.713063 24989 raft_consensus.cc:2454] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:41.713073 24594 raft_consensus.cc:2454] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:41.713318 24791 leader_election.cc:302] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:41.713476 26366 raft_consensus.cc:2781] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.713615 26366 raft_consensus.cc:686] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:41.713780 26366 consensus_queue.cc:237] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:41.715659 24450 catalog_manager.cc:5484] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:41.735219 26678 raft_consensus.cc:482] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.735354 26678 raft_consensus.cc:504] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.735736 26678 leader_election.cc:288] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:41.736171 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56fe9e75a9c418f817185542ae4c1d9" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:41.736189 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56fe9e75a9c418f817185542ae4c1d9" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:41.736315 24989 raft_consensus.cc:2454] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:41.736315 24594 raft_consensus.cc:2454] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:41.736579 24661 leader_election.cc:302] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:41.736708 26678 raft_consensus.cc:2781] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.736760 26678 raft_consensus.cc:482] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.736801 26678 raft_consensus.cc:3037] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.737545 26678 raft_consensus.cc:504] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.737831 26678 leader_election.cc:288] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:41.738121 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56fe9e75a9c418f817185542ae4c1d9" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:41.738122 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56fe9e75a9c418f817185542ae4c1d9" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:41.738209 24986 raft_consensus.cc:3037] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.738219 24594 raft_consensus.cc:3037] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.738906 24594 raft_consensus.cc:2454] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:41.738958 24986 raft_consensus.cc:2454] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:41.739225 24660 leader_election.cc:302] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:41.739385 26678 raft_consensus.cc:2781] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.739470 26678 raft_consensus.cc:686] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:41.739595 26678 consensus_queue.cc:237] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.741503 24450 catalog_manager.cc:5484] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:41.895666 26678 raft_consensus.cc:482] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.895833 26678 raft_consensus.cc:504] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.896270 26678 leader_election.cc:288] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:41.896720 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40a737a78824728b9e85e6a4ded638a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:41.896761 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40a737a78824728b9e85e6a4ded638a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:41.896903 24857 raft_consensus.cc:2454] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:41.896903 24594 raft_consensus.cc:2454] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:41.897284 24661 leader_election.cc:302] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:41.897552 26678 raft_consensus.cc:2781] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.897621 26678 raft_consensus.cc:482] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.897673 26678 raft_consensus.cc:3037] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.898564 26678 raft_consensus.cc:504] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.898933 26678 leader_election.cc:288] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:41.899324 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40a737a78824728b9e85e6a4ded638a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:41.899416 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40a737a78824728b9e85e6a4ded638a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:41.899470 24857 raft_consensus.cc:3037] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.899557 24595 raft_consensus.cc:3037] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.900358 24857 raft_consensus.cc:2454] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:41.900365 24595 raft_consensus.cc:2454] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:41.900713 24661 leader_election.cc:302] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:41.900972 26678 raft_consensus.cc:2781] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.901067 26678 raft_consensus.cc:686] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:41.901227 26678 consensus_queue.cc:237] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:41.903283 24450 catalog_manager.cc:5484] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
W20240501 19:56:42.070603 26699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:42.070907 26699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:42.074318 26699 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:42.094094 26695 consensus_queue.cc:1035] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
I20240501 19:56:42.098309 26695 consensus_queue.cc:1035] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.173573 24989 raft_consensus.cc:1261] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.173839 24594 raft_consensus.cc:1261] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.173981 26678 consensus_queue.cc:1035] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.174227 26677 consensus_queue.cc:1035] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.175719 24594 raft_consensus.cc:1261] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.175948 24857 raft_consensus.cc:1261] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.176306 26677 consensus_queue.cc:1035] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.177481 26677 consensus_queue.cc:1035] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.185770 26736 mvcc.cc:205] Tried to move back new op lower bound from 7022974575299801088 to 7022974574188961792. Current Snapshot: MvccSnapshot[applied={T|T < 7022974575299801088}]
I20240501 19:56:42.186817 26717 mvcc.cc:205] Tried to move back new op lower bound from 7022974575298826240 to 7022974573526536192. Current Snapshot: MvccSnapshot[applied={T|T < 7022974575298826240}]
I20240501 19:56:42.199082 26711 mvcc.cc:205] Tried to move back new op lower bound from 7022974575298826240 to 7022974573526536192. Current Snapshot: MvccSnapshot[applied={T|T < 7022974575298826240}]
INSERT report
rows total: 2000
time total: 112.077 ms
time per row: 0.0560387 ms
I20240501 19:56:42.234299 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:42.240047 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 433347ce828b4d17a70308c421ab4f1f (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.240103 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 433347ce828b4d17a70308c421ab4f1f (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.240373 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433347ce828b4d17a70308c421ab4f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.240373 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433347ce828b4d17a70308c421ab4f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.242672 26758 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:42.243024 24575 tablet_service.cc:1467] Processing CreateTablet for tablet e75026e7e70548e190f81bf79fb30926 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.243068 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 433347ce828b4d17a70308c421ab4f1f (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.243253 24836 tablet_service.cc:1467] Processing CreateTablet for tablet e75026e7e70548e190f81bf79fb30926 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.243253 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1f7372e5ce42b284cf1179c72f578d (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.243320 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75026e7e70548e190f81bf79fb30926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.243320 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433347ce828b4d17a70308c421ab4f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.243497 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75026e7e70548e190f81bf79fb30926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.244659 26758 tablet_bootstrap.cc:654] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.245266 26759 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:42.246162 24707 tablet_service.cc:1467] Processing CreateTablet for tablet e75026e7e70548e190f81bf79fb30926 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.246328 26759 tablet_bootstrap.cc:654] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.246387 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75026e7e70548e190f81bf79fb30926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.246959 26762 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:42.247329 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1f7372e5ce42b284cf1179c72f578d (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.247538 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1f7372e5ce42b284cf1179c72f578d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.247723 26762 tablet_bootstrap.cc:654] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.248486 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1f7372e5ce42b284cf1179c72f578d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.248492 26759 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:42.248580 26759 ts_tablet_manager.cc:1367] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:42.249001 26759 raft_consensus.cc:348] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.249114 26759 raft_consensus.cc:374] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.249157 26759 raft_consensus.cc:729] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.249297 26759 consensus_queue.cc:260] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.250159 26759 ts_tablet_manager.cc:1398] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.250339 26759 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:42.251155 26759 tablet_bootstrap.cc:654] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.252905 26758 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:42.252936 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1f7372e5ce42b284cf1179c72f578d (DEFAULT_TABLE table=rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.253050 26758 ts_tablet_manager.cc:1367] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:42.253180 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1f7372e5ce42b284cf1179c72f578d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.253471 26758 raft_consensus.cc:348] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.253564 26758 raft_consensus.cc:374] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.253598 26758 raft_consensus.cc:729] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.253732 26758 consensus_queue.cc:260] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.253978 26758 ts_tablet_manager.cc:1398] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:42.254141 26758 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:42.254982 26758 tablet_bootstrap.cc:654] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.255932 26762 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:42.256027 26762 ts_tablet_manager.cc:1367] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:42.256399 26758 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:42.256508 26758 ts_tablet_manager.cc:1367] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:42.256898 26758 raft_consensus.cc:348] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.257006 26758 raft_consensus.cc:374] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.257052 26758 raft_consensus.cc:729] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.257158 26758 consensus_queue.cc:260] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.257467 26758 ts_tablet_manager.cc:1398] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.257301 26761 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:42.257628 26758 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:42.258347 26758 tablet_bootstrap.cc:654] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.258435 26761 tablet_bootstrap.cc:654] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.259426 26762 raft_consensus.cc:348] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.259552 26762 raft_consensus.cc:374] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.259588 26762 raft_consensus.cc:729] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.259620 26761 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:42.259697 26761 ts_tablet_manager.cc:1367] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:42.259927 26762 consensus_queue.cc:260] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.260149 26761 raft_consensus.cc:348] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.260241 26762 ts_tablet_manager.cc:1398] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.260253 26761 raft_consensus.cc:374] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.260286 26761 raft_consensus.cc:729] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.260406 26762 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:42.260435 26761 consensus_queue.cc:260] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.260702 26761 ts_tablet_manager.cc:1398] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.261279 26762 tablet_bootstrap.cc:654] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.261971 26759 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:42.262063 26759 ts_tablet_manager.cc:1367] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240501 19:56:42.262445 26759 raft_consensus.cc:348] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.262542 26759 raft_consensus.cc:374] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.262578 26759 raft_consensus.cc:729] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.262722 26759 consensus_queue.cc:260] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.263059 26759 ts_tablet_manager.cc:1398] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:42.263284 26759 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:42.264127 26759 tablet_bootstrap.cc:654] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.264400 26758 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:42.264496 26758 ts_tablet_manager.cc:1367] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:42.265300 26759 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:42.265262 26758 raft_consensus.cc:348] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.265352 26758 raft_consensus.cc:374] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.265381 26759 ts_tablet_manager.cc:1367] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:42.265388 26758 raft_consensus.cc:729] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.265513 26758 consensus_queue.cc:260] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.265796 26759 raft_consensus.cc:348] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.265897 26759 raft_consensus.cc:374] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.265941 26759 raft_consensus.cc:729] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.266065 26759 consensus_queue.cc:260] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.266412 26759 ts_tablet_manager.cc:1398] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:42.267243 26762 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:42.267343 26762 ts_tablet_manager.cc:1367] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:42.267730 26762 raft_consensus.cc:348] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.267840 26762 raft_consensus.cc:374] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.268110 26758 ts_tablet_manager.cc:1398] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:42.268249 26762 raft_consensus.cc:729] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.268402 26762 consensus_queue.cc:260] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.268815 26762 ts_tablet_manager.cc:1398] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.347337 26678 raft_consensus.cc:482] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.347479 26678 raft_consensus.cc:504] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.347879 26678 leader_election.cc:288] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:42.348240 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:42.348249 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:42.348369 24857 raft_consensus.cc:2454] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:42.348388 24989 raft_consensus.cc:2454] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:42.348673 24661 leader_election.cc:302] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:42.348834 26678 raft_consensus.cc:2781] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.348896 26678 raft_consensus.cc:482] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.348942 26678 raft_consensus.cc:3037] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.349779 26678 raft_consensus.cc:504] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.350067 26678 leader_election.cc:288] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:42.350373 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:42.350430 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:42.350476 24857 raft_consensus.cc:3037] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.350549 24989 raft_consensus.cc:3037] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.351253 24857 raft_consensus.cc:2454] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:42.351361 24989 raft_consensus.cc:2454] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:42.351598 24661 leader_election.cc:302] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:42.351739 26678 raft_consensus.cc:2781] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.351827 26678 raft_consensus.cc:686] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:42.351961 26678 consensus_queue.cc:237] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.353785 24449 catalog_manager.cc:5484] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:42.527509 26678 raft_consensus.cc:482] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.527645 26678 raft_consensus.cc:504] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.528060 26678 leader_election.cc:288] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:42.528497 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:42.528513 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:42.528622 24989 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:42.528678 24857 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:42.528940 24661 leader_election.cc:302] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:42.529099 26678 raft_consensus.cc:2781] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.529158 26678 raft_consensus.cc:482] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.529202 26678 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.530123 26678 raft_consensus.cc:504] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.530457 26678 leader_election.cc:288] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:42.530807 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:42.530882 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:42.530926 24857 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.531005 24989 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.531756 24857 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:42.531924 24989 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:42.532181 24661 leader_election.cc:302] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:42.532348 26678 raft_consensus.cc:2781] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.532462 26678 raft_consensus.cc:686] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:42.532591 26678 consensus_queue.cc:237] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.534868 24449 catalog_manager.cc:5484] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:42.608328 26678 raft_consensus.cc:482] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.608510 26678 raft_consensus.cc:504] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.608947 26678 leader_election.cc:288] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:42.609195 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75026e7e70548e190f81bf79fb30926" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:42.609331 24595 raft_consensus.cc:2454] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:42.609259 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75026e7e70548e190f81bf79fb30926" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:42.609546 24857 raft_consensus.cc:2454] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:42.609789 24660 leader_election.cc:302] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:42.609987 26678 raft_consensus.cc:2781] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.610044 26678 raft_consensus.cc:482] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.610097 26678 raft_consensus.cc:3037] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.611091 26678 raft_consensus.cc:504] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.611446 26678 leader_election.cc:288] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:42.611837 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75026e7e70548e190f81bf79fb30926" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:42.611881 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75026e7e70548e190f81bf79fb30926" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:42.611963 24857 raft_consensus.cc:3037] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.611984 24595 raft_consensus.cc:3037] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.612709 24595 raft_consensus.cc:2454] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:42.612725 24857 raft_consensus.cc:2454] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:42.612972 24660 leader_election.cc:302] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:42.613180 26678 raft_consensus.cc:2781] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.613322 26678 raft_consensus.cc:686] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:42.613479 26678 consensus_queue.cc:237] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:42.616510 24449 catalog_manager.cc:5484] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:42.736280 26677 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.740483 26773 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:42.805938 26775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:42.806192 26775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:42.809545 26775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240501 19:56:42.891167 24989 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.891414 24856 raft_consensus.cc:1261] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.892300 26677 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.892442 24595 raft_consensus.cc:1261] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.892751 24856 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.892329 26809 consensus_queue.cc:1035] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.897032 26773 consensus_queue.cc:1035] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.897384 26773 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.930029 26821 mvcc.cc:205] Tried to move back new op lower bound from 7022974578237509632 to 7022974576774942720. Current Snapshot: MvccSnapshot[applied={T|T < 7022974578237509632}]
INSERT report
rows total: 2000
time total: 118.066 ms
time per row: 0.0590332 ms
I20240501 19:56:42.968470 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:42.973855 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 0d91e1c792c749e7883bb996d58d99ea (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.974203 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d91e1c792c749e7883bb996d58d99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.974761 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe0314450814a31ab1bdda90ab245e2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.974934 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe0314450814a31ab1bdda90ab245e2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.975054 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe0314450814a31ab1bdda90ab245e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.974965 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 7beb152e6d5a435eb9c97fa1cace4b39 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.975203 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe0314450814a31ab1bdda90ab245e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.975265 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7beb152e6d5a435eb9c97fa1cace4b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.976673 26830 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:42.974082 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 0d91e1c792c749e7883bb996d58d99ea (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.977336 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 7beb152e6d5a435eb9c97fa1cace4b39 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.977541 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7beb152e6d5a435eb9c97fa1cace4b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.977547 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d91e1c792c749e7883bb996d58d99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.977491 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 0d91e1c792c749e7883bb996d58d99ea (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.977605 26830 tablet_bootstrap.cc:654] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.977747 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d91e1c792c749e7883bb996d58d99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.979629 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe0314450814a31ab1bdda90ab245e2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.979864 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe0314450814a31ab1bdda90ab245e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.980103 26834 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:42.980233 26831 tablet_bootstrap.cc:492] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:42.980286 26832 tablet_bootstrap.cc:492] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:42.980242 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 7beb152e6d5a435eb9c97fa1cace4b39 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.980469 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7beb152e6d5a435eb9c97fa1cace4b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.981009 26834 tablet_bootstrap.cc:654] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.981189 26831 tablet_bootstrap.cc:654] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.981865 26830 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:42.981963 26830 ts_tablet_manager.cc:1367] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:42.982707 26830 raft_consensus.cc:348] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.982821 26830 raft_consensus.cc:374] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.983098 26830 raft_consensus.cc:729] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.983250 26831 tablet_bootstrap.cc:492] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:42.983239 26830 consensus_queue.cc:260] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.983451 26831 ts_tablet_manager.cc:1367] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:42.983536 26830 ts_tablet_manager.cc:1398] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:42.983721 26832 tablet_bootstrap.cc:654] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.983736 26830 tablet_bootstrap.cc:492] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:42.984058 26831 raft_consensus.cc:348] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.984169 26831 raft_consensus.cc:374] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.984206 26831 raft_consensus.cc:729] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.984371 26831 consensus_queue.cc:260] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.984747 26831 ts_tablet_manager.cc:1398] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.984860 26830 tablet_bootstrap.cc:654] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.985976 26830 tablet_bootstrap.cc:492] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:42.986106 26830 ts_tablet_manager.cc:1367] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:42.986157 26832 tablet_bootstrap.cc:492] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:42.986253 26832 ts_tablet_manager.cc:1367] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:42.986598 26830 raft_consensus.cc:348] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.986704 26830 raft_consensus.cc:374] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.986739 26830 raft_consensus.cc:729] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.986670 26832 raft_consensus.cc:348] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.986781 26832 raft_consensus.cc:374] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.986822 26832 raft_consensus.cc:729] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.986842 26830 consensus_queue.cc:260] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.986940 26832 consensus_queue.cc:260] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.987129 26830 ts_tablet_manager.cc:1398] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.987217 26832 ts_tablet_manager.cc:1398] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:42.987334 26830 tablet_bootstrap.cc:492] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:42.987418 26832 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:42.988178 26834 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:42.988260 26834 ts_tablet_manager.cc:1367] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:42.988632 26832 tablet_bootstrap.cc:654] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.988741 26834 raft_consensus.cc:348] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.988858 26834 raft_consensus.cc:374] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.988903 26834 raft_consensus.cc:729] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.988983 26830 tablet_bootstrap.cc:654] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.989037 26834 consensus_queue.cc:260] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.989362 26834 ts_tablet_manager.cc:1398] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.989532 26834 tablet_bootstrap.cc:492] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:42.990433 26834 tablet_bootstrap.cc:654] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.990800 26830 tablet_bootstrap.cc:492] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:42.990869 26830 ts_tablet_manager.cc:1367] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:56:42.991346 26830 raft_consensus.cc:348] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.991457 26830 raft_consensus.cc:374] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.991495 26830 raft_consensus.cc:729] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.991616 26834 tablet_bootstrap.cc:492] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:42.991690 26834 ts_tablet_manager.cc:1367] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.991619 26830 consensus_queue.cc:260] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.992163 26834 raft_consensus.cc:348] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.992269 26834 raft_consensus.cc:374] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.992336 26834 raft_consensus.cc:729] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.992470 26834 consensus_queue.cc:260] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.992718 26834 ts_tablet_manager.cc:1398] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.992920 26830 ts_tablet_manager.cc:1398] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.996480 26832 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:42.996579 26832 ts_tablet_manager.cc:1367] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 19:56:42.997033 26832 raft_consensus.cc:348] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.997156 26832 raft_consensus.cc:374] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.997192 26832 raft_consensus.cc:729] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.997305 26832 consensus_queue.cc:260] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:42.997603 26832 ts_tablet_manager.cc:1398] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:42.997769 26832 tablet_bootstrap.cc:492] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:42.998595 26832 tablet_bootstrap.cc:654] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.999742 26832 tablet_bootstrap.cc:492] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:42.999822 26832 ts_tablet_manager.cc:1367] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:43.000201 26832 raft_consensus.cc:348] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.000289 26832 raft_consensus.cc:374] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.000310 26832 raft_consensus.cc:729] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.000411 26832 consensus_queue.cc:260] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.000623 26832 ts_tablet_manager.cc:1398] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:43.107322 26549 raft_consensus.cc:482] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.107441 26549 raft_consensus.cc:504] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.107847 26549 leader_election.cc:288] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:43.108165 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91e1c792c749e7883bb996d58d99ea" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:43.108203 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91e1c792c749e7883bb996d58d99ea" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:43.108307 24725 raft_consensus.cc:2454] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:43.108307 24986 raft_consensus.cc:2454] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:43.108597 24530 leader_election.cc:302] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:43.108750 26549 raft_consensus.cc:2781] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.108814 26549 raft_consensus.cc:482] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.108853 26549 raft_consensus.cc:3037] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.109781 26549 raft_consensus.cc:504] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.110095 26549 leader_election.cc:288] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:43.110419 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91e1c792c749e7883bb996d58d99ea" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:43.110472 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91e1c792c749e7883bb996d58d99ea" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:43.110534 24725 raft_consensus.cc:3037] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.110569 24989 raft_consensus.cc:3037] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.111495 24725 raft_consensus.cc:2454] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:43.111495 24989 raft_consensus.cc:2454] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:43.111774 24530 leader_election.cc:302] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:43.111925 26549 raft_consensus.cc:2781] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.112008 26549 raft_consensus.cc:686] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:43.112177 26549 consensus_queue.cc:237] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.113912 24450 catalog_manager.cc:5484] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:43.172940 26607 raft_consensus.cc:482] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.173050 26607 raft_consensus.cc:504] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.173400 26607 leader_election.cc:288] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:43.173712 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7beb152e6d5a435eb9c97fa1cace4b39" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:43.173856 24595 raft_consensus.cc:2454] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:43.173833 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7beb152e6d5a435eb9c97fa1cace4b39" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:43.173986 24725 raft_consensus.cc:2454] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:43.174158 24922 leader_election.cc:302] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:43.174360 26607 raft_consensus.cc:2781] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.174422 26607 raft_consensus.cc:482] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.174479 26607 raft_consensus.cc:3037] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.175457 26607 raft_consensus.cc:504] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.175753 26607 leader_election.cc:288] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:43.176056 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7beb152e6d5a435eb9c97fa1cace4b39" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:43.176194 24595 raft_consensus.cc:3037] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.176169 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7beb152e6d5a435eb9c97fa1cace4b39" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:43.176288 24725 raft_consensus.cc:3037] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.177136 24595 raft_consensus.cc:2454] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:43.177238 24725 raft_consensus.cc:2454] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:43.177374 24922 leader_election.cc:302] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:43.177515 26607 raft_consensus.cc:2781] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.177598 26607 raft_consensus.cc:686] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:43.177774 26607 consensus_queue.cc:237] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.180202 24450 catalog_manager.cc:5484] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:43.194330 26607 raft_consensus.cc:482] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.194432 26607 raft_consensus.cc:504] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.194785 26607 leader_election.cc:288] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:43.195112 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe0314450814a31ab1bdda90ab245e2" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:43.195123 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe0314450814a31ab1bdda90ab245e2" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:43.195215 24595 raft_consensus.cc:2454] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:43.195247 24856 raft_consensus.cc:2454] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:43.195430 24922 leader_election.cc:302] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:43.195617 26607 raft_consensus.cc:2781] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.195675 26607 raft_consensus.cc:482] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.195704 26607 raft_consensus.cc:3037] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.196543 26607 raft_consensus.cc:504] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.196863 26607 leader_election.cc:288] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:43.197099 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe0314450814a31ab1bdda90ab245e2" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:43.197144 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe0314450814a31ab1bdda90ab245e2" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:43.197219 24856 raft_consensus.cc:3037] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.197249 24595 raft_consensus.cc:3037] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.197948 24595 raft_consensus.cc:2454] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:43.197965 24856 raft_consensus.cc:2454] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:43.198196 24922 leader_election.cc:302] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:43.198339 26607 raft_consensus.cc:2781] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.198447 26607 raft_consensus.cc:686] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:43.198570 26607 consensus_queue.cc:237] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.200304 24450 catalog_manager.cc:5484] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
W20240501 19:56:43.325521 26846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:43.325773 26846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:43.330040 26846 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240501 19:56:43.426726 24989 raft_consensus.cc:1261] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.427305 24595 raft_consensus.cc:1261] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.427414 26719 consensus_queue.cc:1035] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.427709 26369 consensus_queue.cc:1035] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.428172 24725 raft_consensus.cc:1261] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.428531 24857 raft_consensus.cc:1261] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.428920 26369 consensus_queue.cc:1035] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.428946 26607 consensus_queue.cc:1035] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.429235 24595 raft_consensus.cc:1261] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.429608 26607 consensus_queue.cc:1035] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.433388 24727 raft_consensus.cc:1261] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.433823 26549 consensus_queue.cc:1035] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.437 ms
time per row: 0.0627184 ms
I20240501 19:56:43.509151 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:43.514318 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 9acd3c8577694e799d3f67a0f2f6adcc (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.514329 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 9acd3c8577694e799d3f67a0f2f6adcc (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.514632 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acd3c8577694e799d3f67a0f2f6adcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.514632 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acd3c8577694e799d3f67a0f2f6adcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.514840 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 92dd4bc514f146bb89083fe8a9315308 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.515144 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92dd4bc514f146bb89083fe8a9315308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.515113 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 9acd3c8577694e799d3f67a0f2f6adcc (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.515373 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acd3c8577694e799d3f67a0f2f6adcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.517429 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 783db67a272147ed8752c5d6dba4456e (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.517658 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783db67a272147ed8752c5d6dba4456e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.517653 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 92dd4bc514f146bb89083fe8a9315308 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.517778 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 783db67a272147ed8752c5d6dba4456e (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.517905 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92dd4bc514f146bb89083fe8a9315308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.517905 26892 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:43.517997 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783db67a272147ed8752c5d6dba4456e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.518811 26892 tablet_bootstrap.cc:654] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.519443 26891 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:43.519995 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 92dd4bc514f146bb89083fe8a9315308 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.520195 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92dd4bc514f146bb89083fe8a9315308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.520236 26893 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:43.520315 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 783db67a272147ed8752c5d6dba4456e (DEFAULT_TABLE table=rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.520543 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783db67a272147ed8752c5d6dba4456e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.520551 26891 tablet_bootstrap.cc:654] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.521128 26895 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:43.521234 26893 tablet_bootstrap.cc:654] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.522131 26895 tablet_bootstrap.cc:654] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.522374 26893 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:43.522462 26893 ts_tablet_manager.cc:1367] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:43.522969 26893 raft_consensus.cc:348] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.523190 26893 raft_consensus.cc:374] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.523231 26893 raft_consensus.cc:729] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.523339 26892 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:43.523335 26893 consensus_queue.cc:260] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.523415 26892 ts_tablet_manager.cc:1367] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 19:56:43.523640 26893 ts_tablet_manager.cc:1398] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.523801 26893 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:43.523821 26892 raft_consensus.cc:348] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.523912 26892 raft_consensus.cc:374] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.523941 26892 raft_consensus.cc:729] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.524034 26892 consensus_queue.cc:260] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.524317 26892 ts_tablet_manager.cc:1398] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:43.524466 26892 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:43.524678 26893 tablet_bootstrap.cc:654] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.525350 26892 tablet_bootstrap.cc:654] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.526185 26891 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:43.526288 26891 ts_tablet_manager.cc:1367] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:43.526757 26893 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:43.526832 26893 ts_tablet_manager.cc:1367] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.526806 26891 raft_consensus.cc:348] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.526906 26891 raft_consensus.cc:374] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.526943 26891 raft_consensus.cc:729] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.527081 26891 consensus_queue.cc:260] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.527364 26891 ts_tablet_manager.cc:1398] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.527362 26893 raft_consensus.cc:348] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.527484 26893 raft_consensus.cc:374] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.527519 26893 raft_consensus.cc:729] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.527539 26891 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:43.527629 26893 consensus_queue.cc:260] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.527894 26893 ts_tablet_manager.cc:1398] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.528439 26891 tablet_bootstrap.cc:654] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.529769 26892 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:43.529856 26892 ts_tablet_manager.cc:1367] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 19:56:43.530258 26892 raft_consensus.cc:348] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.530396 26892 raft_consensus.cc:374] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.530462 26892 raft_consensus.cc:729] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.530751 26892 consensus_queue.cc:260] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.531069 26891 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:43.531071 26892 ts_tablet_manager.cc:1398] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:43.531148 26891 ts_tablet_manager.cc:1367] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:43.531678 26895 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:43.531777 26895 ts_tablet_manager.cc:1367] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:43.531834 26891 raft_consensus.cc:348] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.531946 26891 raft_consensus.cc:374] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.531983 26891 raft_consensus.cc:729] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.532101 26891 consensus_queue.cc:260] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.532274 26895 raft_consensus.cc:348] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.532379 26895 raft_consensus.cc:374] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.532398 26891 ts_tablet_manager.cc:1398] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.532420 26895 raft_consensus.cc:729] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.532560 26891 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:43.532541 26895 consensus_queue.cc:260] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.532822 26895 ts_tablet_manager.cc:1398] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.532996 26895 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:43.533567 26891 tablet_bootstrap.cc:654] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.533874 26895 tablet_bootstrap.cc:654] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.537590 26895 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:43.537693 26895 ts_tablet_manager.cc:1367] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:43.537865 26891 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:43.537937 26891 ts_tablet_manager.cc:1367] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:43.538132 26895 raft_consensus.cc:348] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.538240 26895 raft_consensus.cc:374] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.538319 26895 raft_consensus.cc:729] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.538340 26891 raft_consensus.cc:348] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.538450 26891 raft_consensus.cc:374] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.538481 26891 raft_consensus.cc:729] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.538480 26895 consensus_queue.cc:260] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.538578 26369 raft_consensus.cc:482] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.538580 26891 consensus_queue.cc:260] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.538642 26369 raft_consensus.cc:504] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.538870 26891 ts_tablet_manager.cc:1398] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:43.539011 26369 leader_election.cc:288] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:43.539324 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783db67a272147ed8752c5d6dba4456e" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:43.539337 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783db67a272147ed8752c5d6dba4456e" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:43.539453 24856 raft_consensus.cc:2454] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:43.539453 24595 raft_consensus.cc:2454] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:43.539732 24922 leader_election.cc:302] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:43.539866 26369 raft_consensus.cc:2781] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.539909 26369 raft_consensus.cc:482] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.539944 26369 raft_consensus.cc:3037] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.540143 26895 ts_tablet_manager.cc:1398] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:43.540899 26369 raft_consensus.cc:504] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.541247 26369 leader_election.cc:288] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:43.541731 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783db67a272147ed8752c5d6dba4456e" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:43.541839 24595 raft_consensus.cc:3037] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.542608 24595 raft_consensus.cc:2454] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:43.543581 24922 leader_election.cc:302] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:43.543658 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783db67a272147ed8752c5d6dba4456e" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:43.543769 24856 raft_consensus.cc:3037] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.543870 26887 raft_consensus.cc:2781] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.543970 26887 raft_consensus.cc:686] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:43.544080 26887 consensus_queue.cc:237] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:43.544611 24856 raft_consensus.cc:2454] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:43.546101 24445 catalog_manager.cc:5484] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:43.660020 26366 raft_consensus.cc:482] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.660032 26904 raft_consensus.cc:482] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.660159 26366 raft_consensus.cc:504] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.660179 26904 raft_consensus.cc:504] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.660511 26885 raft_consensus.cc:482] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.660645 26366 leader_election.cc:288] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:43.660619 26885 raft_consensus.cc:504] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.660790 26677 raft_consensus.cc:482] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.660928 26677 raft_consensus.cc:504] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.661031 26885 leader_election.cc:288] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:43.661089 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92dd4bc514f146bb89083fe8a9315308" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:43.661110 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92dd4bc514f146bb89083fe8a9315308" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:43.661255 24986 raft_consensus.cc:2454] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:43.661348 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acd3c8577694e799d3f67a0f2f6adcc" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:43.661406 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acd3c8577694e799d3f67a0f2f6adcc" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:43.661458 24725 raft_consensus.cc:2454] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:43.661552 24792 leader_election.cc:302] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:43.661619 24727 raft_consensus.cc:2454] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:43.661875 24791 leader_election.cc:302] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:43.661959 26366 raft_consensus.cc:2781] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.662000 26677 leader_election.cc:288] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:43.662024 26366 raft_consensus.cc:482] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.662070 26366 raft_consensus.cc:3037] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.662235 26906 raft_consensus.cc:2781] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.662297 26906 raft_consensus.cc:482] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.662351 26906 raft_consensus.cc:3037] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.662436 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92dd4bc514f146bb89083fe8a9315308" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:43.662830 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92dd4bc514f146bb89083fe8a9315308" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:43.662953 24986 raft_consensus.cc:2454] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:43.663146 26366 raft_consensus.cc:504] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.663228 24661 leader_election.cc:302] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:43.663367 24857 raft_consensus.cc:2379] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9081292597be4a72a42b216fddcd7971 in current term 1: Already voted for candidate 9d64cdfaeb2d46b29845160e395fc241 in this term.
I20240501 19:56:43.663415 26906 raft_consensus.cc:504] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.663511 26366 leader_election.cc:288] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:43.663664 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acd3c8577694e799d3f67a0f2f6adcc" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:43.663725 26906 leader_election.cc:288] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:43.663782 24856 raft_consensus.cc:2379] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbf7735f82ca4f65a5170ac7909faa47 in current term 1: Already voted for candidate 9d64cdfaeb2d46b29845160e395fc241 in this term.
I20240501 19:56:43.664021 26904 leader_election.cc:288] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:43.664052 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acd3c8577694e799d3f67a0f2f6adcc" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:43.664146 24726 raft_consensus.cc:3037] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.664170 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92dd4bc514f146bb89083fe8a9315308" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:43.664155 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acd3c8577694e799d3f67a0f2f6adcc" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:43.664242 24727 raft_consensus.cc:3037] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.664251 24595 raft_consensus.cc:3037] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.665024 24595 raft_consensus.cc:2454] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:43.665084 24726 raft_consensus.cc:2454] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:43.665148 24727 raft_consensus.cc:2454] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:43.665273 26677 raft_consensus.cc:2741] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 19:56:43.665407 24791 leader_election.cc:302] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:43.665443 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acd3c8577694e799d3f67a0f2f6adcc" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:43.665565 24595 raft_consensus.cc:2362] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1. Re-sending same reply.
I20240501 19:56:43.665629 24791 leader_election.cc:302] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:43.665670 26904 raft_consensus.cc:2781] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.665786 24530 leader_election.cc:302] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:56:43.665822 26906 raft_consensus.cc:2781] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.665874 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92dd4bc514f146bb89083fe8a9315308" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:43.665987 24986 raft_consensus.cc:3037] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.666030 26885 raft_consensus.cc:2726] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:43.666049 26904 raft_consensus.cc:686] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:43.666167 26906 raft_consensus.cc:686] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:43.666194 26904 consensus_queue.cc:237] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:43.666311 26906 consensus_queue.cc:237] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:43.666996 24986 raft_consensus.cc:2454] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:43.668520 24445 catalog_manager.cc:5484] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:43.668746 24445 catalog_manager.cc:5484] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
W20240501 19:56:43.791536 26909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:43.791805 26909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:43.794915 26909 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 19:56:43.885948 24986 raft_consensus.cc:1261] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.886402 26906 consensus_queue.cc:1035] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.886723 24727 raft_consensus.cc:1261] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.887594 26904 consensus_queue.cc:1035] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.888219 24856 raft_consensus.cc:1261] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.888648 26369 consensus_queue.cc:1035] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.898893 24595 raft_consensus.cc:1261] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.899423 26887 consensus_queue.cc:1035] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.906445 24595 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.906580 24726 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.907274 26366 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.907780 26906 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 142.831 ms
time per row: 0.0714156 ms
I20240501 19:56:43.979782 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.5:45247
--local_ip_for_outbound_sockets=127.17.65.5
--tserver_master_addrs=127.17.65.62:40811
--webserver_port=46813
--webserver_interface=127.17.65.5
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:56:44.097541 26950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.097847 26950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:44.097923 26950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.101485 26950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:44.101584 26950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:44.101727 26950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.5
I20240501 19:56:44.105908 26950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.5:45247
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.65.5
--webserver_port=46813
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.26950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.5
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:44.106822 26950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:44.108136 26950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:44.117496 26956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.118156 26958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:44.119263 26950 server_base.cc:1034] running on GCE node
W20240501 19:56:44.120024 26955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:44.120471 26950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:44.121055 26950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:44.122202 26950 hybrid_clock.cc:648] HybridClock initialized: now 1714593404122190 us; error 28 us; skew 500 ppm
I20240501 19:56:44.124138 26950 webserver.cc:460] Webserver started at http://127.17.65.5:46813/ using document root <none> and password file <none>
I20240501 19:56:44.124679 26950 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:44.124756 26950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:44.128065 26950 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:44.129927 26964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.130904 26950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:44.131019 26950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:44.131476 26950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:44.157892 26950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:44.158550 26950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:44.158705 26950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:44.159229 26950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:44.160095 26950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:44.160146 26950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.160203 26950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:44.160228 26950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.168628 26950 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.5:45247
I20240501 19:56:44.168689 27077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.5:45247 every 8 connection(s)
I20240501 19:56:44.169668 26950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:44.176304 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 26950
I20240501 19:56:44.176774 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.6:44579
--local_ip_for_outbound_sockets=127.17.65.6
--tserver_master_addrs=127.17.65.62:40811
--webserver_port=43645
--webserver_interface=127.17.65.6
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:44.178660 27078 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:44.178869 27078 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:44.179311 27078 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:44.180444 24450 ts_manager.cc:194] Re-registered known tserver with Master: 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:44.181247 24450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.5:36263
W20240501 19:56:44.292711 27081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.292930 27081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:44.292971 27081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.295886 27081 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:44.295948 27081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:44.296025 27081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.6
I20240501 19:56:44.299680 27081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.6:44579
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.65.6
--webserver_port=43645
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.27081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.6
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:44.300657 27081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:44.301779 27081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:44.311096 27086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.311955 27089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.312750 27087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:44.313431 27081 server_base.cc:1034] running on GCE node
I20240501 19:56:44.313869 27081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:44.314478 27081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:44.315634 27081 hybrid_clock.cc:648] HybridClock initialized: now 1714593404315614 us; error 53 us; skew 500 ppm
I20240501 19:56:44.317587 27081 webserver.cc:460] Webserver started at http://127.17.65.6:43645/ using document root <none> and password file <none>
I20240501 19:56:44.318121 27081 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:44.318197 27081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:44.321555 27081 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:44.324083 27095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.325139 27081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:44.325251 27081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "29d1a658e2b946e0b412e5fef82db9d9"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:44.325696 27081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:44.356868 27081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:44.357518 27081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:44.357704 27081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:44.358224 27081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:44.359128 27081 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:44.359174 27081 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.359217 27081 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:44.359246 27081 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.368408 27081 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.6:44579
I20240501 19:56:44.368589 27208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.6:44579 every 8 connection(s)
I20240501 19:56:44.369516 27081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:44.374300 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 27081
I20240501 19:56:44.374723 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.7:39797
--local_ip_for_outbound_sockets=127.17.65.7
--tserver_master_addrs=127.17.65.62:40811
--webserver_port=42451
--webserver_interface=127.17.65.7
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:44.380956 27209 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:44.381178 27209 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:44.381628 27209 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:44.382669 24450 ts_manager.cc:194] Re-registered known tserver with Master: 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:44.383481 24450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.6:55451
W20240501 19:56:44.491142 27212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.491343 27212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:44.491385 27212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.494228 27212 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:44.494284 27212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:44.494361 27212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.7
I20240501 19:56:44.498186 27212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.7:39797
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.65.7
--webserver_port=42451
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.27212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.7
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:44.499182 27212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:44.500373 27212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:44.509845 27217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.510497 27220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.511175 27218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:44.512200 27212 server_base.cc:1034] running on GCE node
I20240501 19:56:44.512567 27212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:44.513166 27212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:44.514320 27212 hybrid_clock.cc:648] HybridClock initialized: now 1714593404514303 us; error 50 us; skew 500 ppm
I20240501 19:56:44.516350 27212 webserver.cc:460] Webserver started at http://127.17.65.7:42451/ using document root <none> and password file <none>
I20240501 19:56:44.516903 27212 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:44.516984 27212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:44.520306 27212 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:44.522131 27226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.523180 27212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:44.523284 27212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "53e34bb09d8e41129a5683079ff63610"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:44.523732 27212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:44.563854 27212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:44.564579 27212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:44.564744 27212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:44.565375 27212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:44.566439 27212 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:44.566522 27212 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.566571 27212 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:44.566598 27212 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.576303 27212 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.7:39797
I20240501 19:56:44.576339 27339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.7:39797 every 8 connection(s)
I20240501 19:56:44.577543 27212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:44.581238 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 27212
I20240501 19:56:44.581652 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.8:35957
--local_ip_for_outbound_sockets=127.17.65.8
--tserver_master_addrs=127.17.65.62:40811
--webserver_port=34553
--webserver_interface=127.17.65.8
--builtin_ntp_servers=127.17.65.20:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:44.586714 27340 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:44.586933 27340 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:44.587415 27340 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:44.588346 24450 ts_manager.cc:194] Re-registered known tserver with Master: 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:44.588900 24450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.7:54351
W20240501 19:56:44.693897 27343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.694111 27343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:44.694159 27343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.697227 27343 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:44.697292 27343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:44.697371 27343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.8
I20240501 19:56:44.701186 27343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.8:35957
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.17.65.8
--webserver_port=34553
--tserver_master_addrs=127.17.65.62:40811
--never_fsync=true
--heap_profile_path=/tmp/kudu.27343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.8
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:44.702103 27343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:44.703270 27343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:44.711802 27348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.712435 27349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.713389 27351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:44.714259 27343 server_base.cc:1034] running on GCE node
I20240501 19:56:44.714665 27343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:44.715287 27343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:44.716454 27343 hybrid_clock.cc:648] HybridClock initialized: now 1714593404716435 us; error 38 us; skew 500 ppm
I20240501 19:56:44.718389 27343 webserver.cc:460] Webserver started at http://127.17.65.8:34553/ using document root <none> and password file <none>
I20240501 19:56:44.718962 27343 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:44.719076 27343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:44.722882 27343 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 19:56:44.724881 27358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.725891 27343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:44.726009 27343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4a6e54e86a9949e5970b42e8e32e7561"
format_stamp: "Formatted at 2024-05-01 19:56:28 on dist-test-slave-703j"
I20240501 19:56:44.726414 27343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:44.744036 27343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:44.744731 27343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:44.744889 27343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:44.745450 27343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:44.746333 27343 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:44.746384 27343 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.746449 27343 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:44.746490 27343 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.757757 27343 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.8:35957
I20240501 19:56:44.757793 27471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.8:35957 every 8 connection(s)
I20240501 19:56:44.759121 27343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:44.768102 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 27343
I20240501 19:56:44.771272 27472 heartbeater.cc:347] Connected to a master server at 127.17.65.62:40811
I20240501 19:56:44.771530 27472 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:44.772131 27472 heartbeater.cc:511] Master 127.17.65.62:40811 requested a full tablet report, sending...
I20240501 19:56:44.772734 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:44.773123 24454 ts_manager.cc:194] Re-registered known tserver with Master: 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:44.773692 24454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.8:48839
I20240501 19:56:44.788775 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4ddebc1797408c9d45e086c9365952 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77]), partition=
I20240501 19:56:44.789081 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4ddebc1797408c9d45e086c9365952 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77]), partition=
I20240501 19:56:44.789470 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4ddebc1797408c9d45e086c9365952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.789696 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4ddebc1797408c9d45e086c9365952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.795279 27489 tablet_bootstrap.cc:492] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:44.796085 27490 tablet_bootstrap.cc:492] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:44.797884 27490 tablet_bootstrap.cc:654] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.797884 27489 tablet_bootstrap.cc:654] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.798712 27490 log.cc:824] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:44.799456 27489 log.cc:824] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:44.800352 27490 tablet_bootstrap.cc:492] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:44.800529 27490 ts_tablet_manager.cc:1367] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 19:56:44.804529 27490 raft_consensus.cc:348] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.804746 27490 raft_consensus.cc:374] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.804808 27490 raft_consensus.cc:729] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.805236 27490 consensus_queue.cc:260] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.806118 27490 ts_tablet_manager.cc:1398] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240501 19:56:44.806505 27472 heartbeater.cc:503] Master 127.17.65.62:40811 was elected leader, sending a full tablet report...
I20240501 19:56:44.809504 27489 tablet_bootstrap.cc:492] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:44.809682 27489 ts_tablet_manager.cc:1367] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 19:56:44.812618 27489 raft_consensus.cc:348] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.812784 27489 raft_consensus.cc:374] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.812848 27489 raft_consensus.cc:729] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.813241 27489 consensus_queue.cc:260] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.813814 27209 heartbeater.cc:503] Master 127.17.65.62:40811 was elected leader, sending a full tablet report...
I20240501 19:56:44.814020 27489 ts_tablet_manager.cc:1398] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:44.817278 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4ddebc1797408c9d45e086c9365952 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77]), partition=
I20240501 19:56:44.817974 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4ddebc1797408c9d45e086c9365952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.824632 27496 tablet_bootstrap.cc:492] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:44.826745 27496 tablet_bootstrap.cc:654] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.827488 27496 log.cc:824] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:44.828979 27496 tablet_bootstrap.cc:492] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:44.829180 27496 ts_tablet_manager.cc:1367] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:56:44.832012 27496 raft_consensus.cc:348] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.832549 27496 raft_consensus.cc:374] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.832613 27496 raft_consensus.cc:729] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.833012 27496 consensus_queue.cc:260] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.833644 27078 heartbeater.cc:503] Master 127.17.65.62:40811 was elected leader, sending a full tablet report...
I20240501 19:56:44.833796 27496 ts_tablet_manager.cc:1398] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20240501 19:56:44.870537 27210 tablet.cc:2369] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:44.900709 27494 raft_consensus.cc:482] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.900911 27494 raft_consensus.cc:504] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.902029 27494 leader_election.cc:288] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:44.906850 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ddebc1797408c9d45e086c9365952" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:44.907135 27163 raft_consensus.cc:2454] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:44.907425 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ddebc1797408c9d45e086c9365952" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:44.907682 27032 raft_consensus.cc:2454] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:44.907682 27361 leader_election.cc:302] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:44.907928 27494 raft_consensus.cc:2781] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.907989 27494 raft_consensus.cc:482] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.908025 27494 raft_consensus.cc:3037] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.909096 27494 raft_consensus.cc:504] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.909456 27494 leader_election.cc:288] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:44.909799 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ddebc1797408c9d45e086c9365952" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:44.909799 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ddebc1797408c9d45e086c9365952" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:44.909914 27163 raft_consensus.cc:3037] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.909914 27032 raft_consensus.cc:3037] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.910984 27163 raft_consensus.cc:2454] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:44.910984 27032 raft_consensus.cc:2454] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:44.911324 27359 leader_election.cc:302] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:44.911629 27494 raft_consensus.cc:2781] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.911831 27494 raft_consensus.cc:686] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:44.912163 27494 consensus_queue.cc:237] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:44.915305 24450 catalog_manager.cc:5484] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
W20240501 19:56:44.920892 27079 tablet.cc:2369] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:45.010186 27473 tablet.cc:2369] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:45.031136 27012 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.034344 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:45.042025 27032 raft_consensus.cc:1261] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.042919 27503 consensus_queue.cc:1035] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.043172 27143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.043200 27163 raft_consensus.cc:1261] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.043944 27503 consensus_queue.cc:1035] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.044881 24837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:45.046911 24706 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:56:45.048631 24451 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 1c4ddebc1797408c9d45e086c9365952 (table rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77]). Expected version 1 got 0
I20240501 19:56:45.049677 24451 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 1c4ddebc1797408c9d45e086c9365952 (table rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77]). Expected version 1 got 0
I20240501 19:56:45.050748 27563 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:45.051622 27563 tablet.cc:1620] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:45.054116 24454 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 1c4ddebc1797408c9d45e086c9365952 (table rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77]). Expected version 1 got 0
I20240501 19:56:45.054387 27565 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:45.055016 24447 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 1c4ddebc1797408c9d45e086c9365952 (table rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77]). Expected version 1 got 0
I20240501 19:56:45.055299 27565 tablet.cc:1620] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:45.056746 27559 mvcc.cc:205] Tried to move back new op lower bound from 7022974587044741120 to 7022974586522886144. Current Snapshot: MvccSnapshot[applied={T|T < 7022974587044741120}]
I20240501 19:56:45.057969 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77] alter complete (version 1)
I20240501 19:56:45.060031 24967 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:45.061491 27406 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:45.061882 24574 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:45.076980 27564 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:45.077894 27564 tablet.cc:1620] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:45.106657 27267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.162226 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:45.167459 27563 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:45.167806 27563 tablet.cc:1620] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:45.167858 27565 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:45.168129 27565 tablet.cc:1620] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:45.172184 27564 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:45.172570 27564 tablet.cc:1620] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:45.174530 24451 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=0ba845b31ab64eb7b61d85c5241a2a77] alter complete (version 2)
I20240501 19:56:45.243458 27143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.249516 27406 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:45.254365 27267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.261684 24706 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:56:45.267637 27012 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.269260 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 19:56:45.269430 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 19:56:45.273947 24445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:45.275338 24574 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:45.275692 24837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:45.276477 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 4f2964308d1543f081c2c02ca51eb0e1
I20240501 19:56:45.276849 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet a4f17d0a3d0b49beaa435f32bc50530f
I20240501 19:56:45.277022 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet e8dde944a715448eac832f3e74e45138
I20240501 19:56:45.277202 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2964308d1543f081c2c02ca51eb0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:45.277949 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet e8dde944a715448eac832f3e74e45138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:45.278033 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2964308d1543f081c2c02ca51eb0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:45.278427 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f17d0a3d0b49beaa435f32bc50530f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:45.278475 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f17d0a3d0b49beaa435f32bc50530f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:45.278769 24967 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2964308d1543f081c2c02ca51eb0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:45.278709 27624 tablet_replica.cc:330] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:45.278769 24969 tablet_service.cc:1514] Processing DeleteTablet for tablet e8dde944a715448eac832f3e74e45138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:45.279208 27624 raft_consensus.cc:2229] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.279419 27624 raft_consensus.cc:2258] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.279493 27625 tablet_replica.cc:330] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:45.279812 27625 raft_consensus.cc:2229] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.280040 27625 raft_consensus.cc:2258] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.280622 27624 ts_tablet_manager.cc:1875] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.282166 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f17d0a3d0b49beaa435f32bc50530f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:45.282404 27625 ts_tablet_manager.cc:1875] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.282467 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet e8dde944a715448eac832f3e74e45138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:45.284060 27624 ts_tablet_manager.cc:1888] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.284189 27624 log.cc:1196] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f2964308d1543f081c2c02ca51eb0e1
I20240501 19:56:45.284621 27624 ts_tablet_manager.cc:1909] T 4f2964308d1543f081c2c02ca51eb0e1 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:45.284722 27625 ts_tablet_manager.cc:1888] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.284853 27625 log.cc:1196] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/4f2964308d1543f081c2c02ca51eb0e1
I20240501 19:56:45.285203 27625 ts_tablet_manager.cc:1909] T 4f2964308d1543f081c2c02ca51eb0e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:45.285667 27626 tablet_replica.cc:330] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:45.285825 27624 tablet_replica.cc:330] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:45.286013 27624 raft_consensus.cc:2229] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.286074 27626 raft_consensus.cc:2229] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.286418 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 4f2964308d1543f081c2c02ca51eb0e1 (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.286659 27625 tablet_replica.cc:330] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:45.286602 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 56017e72b6af48ccab39219fb7196bee (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.286782 27625 raft_consensus.cc:2229] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.286768 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 4f2964308d1543f081c2c02ca51eb0e1 (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.286864 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56017e72b6af48ccab39219fb7196bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.286899 27625 raft_consensus.cc:2258] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.287305 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 56017e72b6af48ccab39219fb7196bee (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.287277 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 51cd33a27a0a4754bbe044b0ff0ef262 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.287559 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56017e72b6af48ccab39219fb7196bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.287752 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cd33a27a0a4754bbe044b0ff0ef262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.288527 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 51cd33a27a0a4754bbe044b0ff0ef262 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.288856 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cd33a27a0a4754bbe044b0ff0ef262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.289337 27624 raft_consensus.cc:2258] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.289871 27624 ts_tablet_manager.cc:1875] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.290277 27011 tablet_service.cc:1467] Processing CreateTablet for tablet caf81a72d0304b8aa611d4c1eab5ac26 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.290517 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf81a72d0304b8aa611d4c1eab5ac26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.291138 27625 ts_tablet_manager.cc:1875] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.291353 27142 tablet_service.cc:1467] Processing CreateTablet for tablet caf81a72d0304b8aa611d4c1eab5ac26 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.291625 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf81a72d0304b8aa611d4c1eab5ac26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.291780 27489 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:45.292692 27489 tablet_bootstrap.cc:654] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.293113 27625 ts_tablet_manager.cc:1888] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.293180 27625 log.cc:1196] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/e8dde944a715448eac832f3e74e45138
I20240501 19:56:45.293437 27625 ts_tablet_manager.cc:1909] T e8dde944a715448eac832f3e74e45138 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:45.293949 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 51cd33a27a0a4754bbe044b0ff0ef262 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.294041 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet e8dde944a715448eac832f3e74e45138 (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.294178 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cd33a27a0a4754bbe044b0ff0ef262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.294309 27628 tablet_bootstrap.cc:492] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:45.295150 27628 tablet_bootstrap.cc:654] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.296221 27628 tablet_bootstrap.cc:492] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:45.296296 27628 ts_tablet_manager.cc:1367] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:45.296439 24574 tablet_service.cc:1467] Processing CreateTablet for tablet caf81a72d0304b8aa611d4c1eab5ac26 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.296650 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf81a72d0304b8aa611d4c1eab5ac26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.296741 27628 raft_consensus.cc:348] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.296864 27628 raft_consensus.cc:374] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.296898 27628 raft_consensus.cc:729] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.297029 27628 consensus_queue.cc:260] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.297367 27628 ts_tablet_manager.cc:1398] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:45.299170 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 56017e72b6af48ccab39219fb7196bee (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.300084 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56017e72b6af48ccab39219fb7196bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.301844 27496 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:45.302598 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:45.302862 27496 tablet_bootstrap.cc:654] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.304172 27626 raft_consensus.cc:2258] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.305330 27626 ts_tablet_manager.cc:1875] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.306982 27631 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:45.308291 27631 tablet_bootstrap.cc:654] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.309379 27626 ts_tablet_manager.cc:1888] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.309998 27626 log.cc:1196] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/a4f17d0a3d0b49beaa435f32bc50530f
I20240501 19:56:45.310398 27626 ts_tablet_manager.cc:1909] T a4f17d0a3d0b49beaa435f32bc50530f P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:45.310415 27624 ts_tablet_manager.cc:1888] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.310508 27624 log.cc:1196] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8dde944a715448eac832f3e74e45138
I20240501 19:56:45.310791 27624 ts_tablet_manager.cc:1909] T e8dde944a715448eac832f3e74e45138 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:45.311419 27624 tablet_replica.cc:330] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:45.311633 27626 tablet_replica.cc:330] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:45.311697 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet a4f17d0a3d0b49beaa435f32bc50530f (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.311864 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet e8dde944a715448eac832f3e74e45138 (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.311861 27624 raft_consensus.cc:2229] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.311990 27626 raft_consensus.cc:2229] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.312000 27624 raft_consensus.cc:2258] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.312240 27626 raft_consensus.cc:2258] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.312355 27563 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:45.312474 27624 ts_tablet_manager.cc:1875] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.312920 27626 ts_tablet_manager.cc:1875] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.313206 27489 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:45.313308 27489 ts_tablet_manager.cc:1367] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20240501 19:56:45.314093 27489 raft_consensus.cc:348] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.314224 27489 raft_consensus.cc:374] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.314280 27489 raft_consensus.cc:729] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.314407 27489 consensus_queue.cc:260] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.314702 27489 ts_tablet_manager.cc:1398] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.314893 27489 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:45.315644 27626 ts_tablet_manager.cc:1888] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.315629 27565 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:45.315809 27626 log.cc:1196] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/e8dde944a715448eac832f3e74e45138
I20240501 19:56:45.315910 27489 tablet_bootstrap.cc:654] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.316174 27626 ts_tablet_manager.cc:1909] T e8dde944a715448eac832f3e74e45138 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:45.316851 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet e8dde944a715448eac832f3e74e45138 (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.318313 27564 tablet.cc:1722] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:45.311699 27627 tablet_bootstrap.cc:492] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:45.319628 27627 tablet_bootstrap.cc:654] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.320495 27633 tablet_bootstrap.cc:492] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:45.323160 27624 ts_tablet_manager.cc:1888] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.323542 24969 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:45.323762 27624 log.cc:1196] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4f17d0a3d0b49beaa435f32bc50530f
I20240501 19:56:45.324028 27624 ts_tablet_manager.cc:1909] T a4f17d0a3d0b49beaa435f32bc50530f P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:45.324640 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet a4f17d0a3d0b49beaa435f32bc50530f (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.325353 27633 tablet_bootstrap.cc:654] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.325781 27489 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:45.325878 27489 ts_tablet_manager.cc:1367] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240501 19:56:45.326318 27489 raft_consensus.cc:348] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.326418 27489 raft_consensus.cc:374] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.326458 27489 raft_consensus.cc:729] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.326614 27489 consensus_queue.cc:260] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.326860 27489 ts_tablet_manager.cc:1398] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240501 19:56:45.330729 24791 consensus_peers.cc:485] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f2964308d1543f081c2c02ca51eb0e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.332976 27633 log.cc:824] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:45.333863 27627 tablet_bootstrap.cc:492] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:45.334033 27627 ts_tablet_manager.cc:1367] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 19:56:45.334566 27627 raft_consensus.cc:348] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.334769 27627 raft_consensus.cc:374] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.334849 27627 raft_consensus.cc:729] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.335013 27627 consensus_queue.cc:260] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.335366 27627 ts_tablet_manager.cc:1398] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:45.337600 27633 tablet_bootstrap.cc:492] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:45.337854 27633 ts_tablet_manager.cc:1367] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240501 19:56:45.331173 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=0ba845b31ab64eb7b61d85c5241a2a77] alter complete (version 3)
I20240501 19:56:45.342298 27633 raft_consensus.cc:348] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.342649 27633 raft_consensus.cc:374] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.342756 27633 raft_consensus.cc:729] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.343366 27633 consensus_queue.cc:260] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.344352 27633 ts_tablet_manager.cc:1398] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240501 19:56:45.344693 27340 heartbeater.cc:503] Master 127.17.65.62:40811 was elected leader, sending a full tablet report...
I20240501 19:56:45.345721 27639 raft_consensus.cc:482] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.345853 27639 raft_consensus.cc:504] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.348162 27639 leader_election.cc:288] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:45.352120 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56017e72b6af48ccab39219fb7196bee" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:45.352270 24725 raft_consensus.cc:2454] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:45.352627 27227 leader_election.cc:302] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:45.352957 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56017e72b6af48ccab39219fb7196bee" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:45.353078 24857 raft_consensus.cc:2454] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:45.353574 27639 raft_consensus.cc:2781] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.353667 27639 raft_consensus.cc:482] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.353709 27639 raft_consensus.cc:3037] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:56:45.354537 24791 consensus_peers.cc:485] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4f17d0a3d0b49beaa435f32bc50530f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.354856 27639 raft_consensus.cc:504] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.355516 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56017e72b6af48ccab39219fb7196bee" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:45.355506 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56017e72b6af48ccab39219fb7196bee" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:45.355625 24857 raft_consensus.cc:3037] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.355633 24725 raft_consensus.cc:3037] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.356436 24857 raft_consensus.cc:2454] T 56017e72b6af48ccab39219fb7196bee P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:45.356619 27639 leader_election.cc:288] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:45.356640 24725 raft_consensus.cc:2454] T 56017e72b6af48ccab39219fb7196bee P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:45.356750 27228 leader_election.cc:302] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:45.357033 27639 raft_consensus.cc:2781] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.357221 27639 raft_consensus.cc:686] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:45.357539 27639 consensus_queue.cc:237] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:45.361140 24454 catalog_manager.cc:5484] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:45.367105 27496 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:45.367210 27496 ts_tablet_manager.cc:1367] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.002s
I20240501 19:56:45.367745 27496 raft_consensus.cc:348] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.367846 27496 raft_consensus.cc:374] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.367872 27496 raft_consensus.cc:729] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.367985 27496 consensus_queue.cc:260] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.368237 27496 ts_tablet_manager.cc:1398] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:45.368410 27496 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:45.369226 27496 tablet_bootstrap.cc:654] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.370267 27496 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:45.370343 27496 ts_tablet_manager.cc:1367] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:45.370702 27496 raft_consensus.cc:348] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.370800 27496 raft_consensus.cc:374] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.370827 27496 raft_consensus.cc:729] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.370962 27496 consensus_queue.cc:260] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.371268 27496 ts_tablet_manager.cc:1398] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:45.374963 27623 tablet_replica.cc:330] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:45.375325 27623 raft_consensus.cc:2229] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.375761 27623 raft_consensus.cc:2258] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.375797 27631 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:45.375897 27631 ts_tablet_manager.cc:1367] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.000s
I20240501 19:56:45.376348 27631 raft_consensus.cc:348] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.376467 27631 raft_consensus.cc:374] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.376509 27631 raft_consensus.cc:729] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.376626 27631 consensus_queue.cc:260] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.376920 27631 ts_tablet_manager.cc:1398] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.377067 27631 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:45.377140 27623 ts_tablet_manager.cc:1875] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.377892 27631 tablet_bootstrap.cc:654] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.380192 27623 ts_tablet_manager.cc:1888] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.380336 27623 log.cc:1196] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/4f2964308d1543f081c2c02ca51eb0e1
I20240501 19:56:45.381270 27623 ts_tablet_manager.cc:1909] T 4f2964308d1543f081c2c02ca51eb0e1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:45.383671 27623 tablet_replica.cc:330] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:45.383809 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 4f2964308d1543f081c2c02ca51eb0e1 (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.384296 27623 raft_consensus.cc:2229] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.384546 27631 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:45.385013 27623 raft_consensus.cc:2258] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.384846 27631 ts_tablet_manager.cc:1367] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:45.385576 27623 ts_tablet_manager.cc:1875] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.385579 27631 raft_consensus.cc:348] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.386372 27631 raft_consensus.cc:374] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.386492 27631 raft_consensus.cc:729] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.386710 27631 consensus_queue.cc:260] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.387161 27631 ts_tablet_manager.cc:1398] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:45.387658 27623 ts_tablet_manager.cc:1888] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.387737 27623 log.cc:1196] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/a4f17d0a3d0b49beaa435f32bc50530f
I20240501 19:56:45.388082 27623 ts_tablet_manager.cc:1909] T a4f17d0a3d0b49beaa435f32bc50530f P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:45.388762 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet a4f17d0a3d0b49beaa435f32bc50530f (table rebalance_test_table_0 [id=25db57eb428c4bcf8346fe47d53ad59a]) successfully deleted
I20240501 19:56:45.397902 26885 raft_consensus.cc:482] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.398005 26885 raft_consensus.cc:504] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.398465 26885 leader_election.cc:288] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:45.404922 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf81a72d0304b8aa611d4c1eab5ac26" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:45.405069 27163 raft_consensus.cc:2454] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:45.405403 24531 leader_election.cc:302] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:45.405565 26885 raft_consensus.cc:2781] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.405615 26885 raft_consensus.cc:482] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.405671 26885 raft_consensus.cc:3037] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.406580 26885 raft_consensus.cc:504] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.406965 26885 leader_election.cc:288] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:45.407992 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf81a72d0304b8aa611d4c1eab5ac26" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:45.408125 27163 raft_consensus.cc:3037] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.409087 27163 raft_consensus.cc:2454] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:45.409214 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf81a72d0304b8aa611d4c1eab5ac26" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:45.409350 27032 raft_consensus.cc:2454] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:45.409566 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf81a72d0304b8aa611d4c1eab5ac26" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:45.409685 27031 raft_consensus.cc:3037] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.410497 27031 raft_consensus.cc:2454] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:45.411005 24531 leader_election.cc:302] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:45.411418 26885 raft_consensus.cc:2781] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.411501 26885 raft_consensus.cc:686] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:45.411634 26885 consensus_queue.cc:237] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:45.413780 24454 catalog_manager.cc:5484] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:45.435241 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:45.435423 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:45.437559 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 1c4ddebc1797408c9d45e086c9365952
I20240501 19:56:45.438342 27406 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4ddebc1797408c9d45e086c9365952 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:45.438504 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4ddebc1797408c9d45e086c9365952 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:45.438663 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4ddebc1797408c9d45e086c9365952 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:45.439158 27666 tablet_replica.cc:330] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:45.439647 27665 tablet_replica.cc:330] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:45.440033 27665 raft_consensus.cc:2229] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.440323 27665 raft_consensus.cc:2258] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.440879 27666 raft_consensus.cc:2229] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.441115 27664 tablet_replica.cc:330] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:45.441298 27666 raft_consensus.cc:2258] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.441367 27664 raft_consensus.cc:2229] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.441710 27664 raft_consensus.cc:2258] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.441718 27665 ts_tablet_manager.cc:1875] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.442569 27664 ts_tablet_manager.cc:1875] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.442801 27666 ts_tablet_manager.cc:1875] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.445179 27664 ts_tablet_manager.cc:1888] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:45.445279 27664 log.cc:1196] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/1c4ddebc1797408c9d45e086c9365952
I20240501 19:56:45.446162 27665 ts_tablet_manager.cc:1888] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:45.446285 27665 log.cc:1196] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/1c4ddebc1797408c9d45e086c9365952
I20240501 19:56:45.446357 27664 ts_tablet_manager.cc:1909] T 1c4ddebc1797408c9d45e086c9365952 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:45.446424 27666 ts_tablet_manager.cc:1888] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:45.446586 27666 log.cc:1196] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/1c4ddebc1797408c9d45e086c9365952
I20240501 19:56:45.446705 27665 ts_tablet_manager.cc:1909] T 1c4ddebc1797408c9d45e086c9365952 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:45.447022 27666 ts_tablet_manager.cc:1909] T 1c4ddebc1797408c9d45e086c9365952 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:45.447594 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 1c4ddebc1797408c9d45e086c9365952 (table rebalancer_dynamic_table_new_name [id=0ba845b31ab64eb7b61d85c5241a2a77]) successfully deleted
I20240501 19:56:45.447803 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 1c4ddebc1797408c9d45e086c9365952 (table rebalancer_dynamic_table_new_name [id=0ba845b31ab64eb7b61d85c5241a2a77]) successfully deleted
I20240501 19:56:45.448146 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 1c4ddebc1797408c9d45e086c9365952 (table rebalancer_dynamic_table_new_name [id=0ba845b31ab64eb7b61d85c5241a2a77]) successfully deleted
I20240501 19:56:45.468566 27498 raft_consensus.cc:482] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.468740 27498 raft_consensus.cc:504] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.469817 27498 leader_election.cc:288] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:45.473834 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cd33a27a0a4754bbe044b0ff0ef262" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:45.474025 27163 raft_consensus.cc:2454] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:45.474277 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cd33a27a0a4754bbe044b0ff0ef262" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:45.474416 24595 raft_consensus.cc:2454] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:45.474520 26967 leader_election.cc:302] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:56:45.474766 27498 raft_consensus.cc:2781] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.474835 27498 raft_consensus.cc:482] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.474880 27498 raft_consensus.cc:3037] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.475991 27498 raft_consensus.cc:504] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.476346 27498 leader_election.cc:288] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:45.476660 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cd33a27a0a4754bbe044b0ff0ef262" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:45.476787 24595 raft_consensus.cc:3037] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.476948 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cd33a27a0a4754bbe044b0ff0ef262" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:45.477070 27163 raft_consensus.cc:3037] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.477844 24595 raft_consensus.cc:2454] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:45.478046 27163 raft_consensus.cc:2454] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:45.478142 26965 leader_election.cc:302] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:45.478322 27498 raft_consensus.cc:2781] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.478499 27498 raft_consensus.cc:686] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:45.478749 27498 consensus_queue.cc:237] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.481272 24450 catalog_manager.cc:5484] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:45.534379 27406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.539218 27012 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:45.539857 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:45.545929 24707 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:56:45.548116 27142 tablet_service.cc:1467] Processing CreateTablet for tablet fb96a28495e549b1b4264eaa08611265 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]), partition=
I20240501 19:56:45.548116 24574 tablet_service.cc:1467] Processing CreateTablet for tablet fb96a28495e549b1b4264eaa08611265 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]), partition=
I20240501 19:56:45.548360 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb96a28495e549b1b4264eaa08611265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.548360 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb96a28495e549b1b4264eaa08611265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.548559 24969 tablet_service.cc:1467] Processing CreateTablet for tablet fb96a28495e549b1b4264eaa08611265 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]), partition=
I20240501 19:56:45.548810 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb96a28495e549b1b4264eaa08611265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.550467 27489 tablet_bootstrap.cc:492] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:45.550520 27631 tablet_bootstrap.cc:492] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:45.551376 27631 tablet_bootstrap.cc:654] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.551425 27489 tablet_bootstrap.cc:654] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.551685 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.552120 27267 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:45.554986 27708 tablet_bootstrap.cc:492] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:45.555943 24837 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:45.559461 24969 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:45.565575 27631 tablet_bootstrap.cc:492] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:45.565696 27631 ts_tablet_manager.cc:1367] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240501 19:56:45.566133 27631 raft_consensus.cc:348] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.566237 27631 raft_consensus.cc:374] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.566267 27631 raft_consensus.cc:729] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.566370 27631 consensus_queue.cc:260] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.566664 27631 ts_tablet_manager.cc:1398] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.570119 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:56:45.574792 27489 tablet_bootstrap.cc:492] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:45.574925 27489 ts_tablet_manager.cc:1367] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s
I20240501 19:56:45.575371 27489 raft_consensus.cc:348] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.575491 27489 raft_consensus.cc:374] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.575525 27489 raft_consensus.cc:729] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.575649 27489 consensus_queue.cc:260] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.575917 27489 ts_tablet_manager.cc:1398] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240501 19:56:45.578960 27341 tablet.cc:2369] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:45.597887 27708 tablet_bootstrap.cc:654] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.612848 27708 tablet_bootstrap.cc:492] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:45.612967 27708 ts_tablet_manager.cc:1367] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.002s
I20240501 19:56:45.613413 27708 raft_consensus.cc:348] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.613529 27708 raft_consensus.cc:374] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.613559 27708 raft_consensus.cc:729] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.615496 27708 consensus_queue.cc:260] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.615852 27708 ts_tablet_manager.cc:1398] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:45.646255 26607 raft_consensus.cc:482] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.646354 26607 raft_consensus.cc:504] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.646831 26607 leader_election.cc:288] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:45.647255 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb96a28495e549b1b4264eaa08611265" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:45.647397 24594 raft_consensus.cc:2454] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:45.647711 24922 leader_election.cc:302] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:45.647939 26607 raft_consensus.cc:2781] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.648005 26607 raft_consensus.cc:482] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.648052 26607 raft_consensus.cc:3037] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.648947 26607 raft_consensus.cc:504] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.649328 26607 leader_election.cc:288] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:45.649698 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb96a28495e549b1b4264eaa08611265" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:45.649812 24594 raft_consensus.cc:3037] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.650714 24594 raft_consensus.cc:2454] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:45.651005 24922 leader_election.cc:302] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:45.651216 26607 raft_consensus.cc:2781] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.651381 26607 raft_consensus.cc:686] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:45.651530 26607 consensus_queue.cc:237] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.651854 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb96a28495e549b1b4264eaa08611265" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:45.651892 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb96a28495e549b1b4264eaa08611265" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:45.651985 27163 raft_consensus.cc:2454] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:45.654296 24450 catalog_manager.cc:5484] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:45.667208 24857 consensus_queue.cc:237] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:45.668897 24725 raft_consensus.cc:1261] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.669157 24986 raft_consensus.cc:1261] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.669576 27484 consensus_queue.cc:1035] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.669780 27731 consensus_queue.cc:1035] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.672345 27484 raft_consensus.cc:2932] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:45.672819 24725 raft_consensus.cc:2932] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
W20240501 19:56:45.674381 24793 consensus_peers.cc:485] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f2d49dcdd8f49d48d8a9bd6d10c332c. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.674875 24986 raft_consensus.cc:2932] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:45.675829 24450 catalog_manager.cc:5484] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:45.696303 24986 consensus_queue.cc:237] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:45.697751 24857 raft_consensus.cc:1261] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.697742 24725 raft_consensus.cc:1261] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.698181 27751 consensus_queue.cc:1035] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.698498 26607 consensus_queue.cc:1035] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.700430 27751 raft_consensus.cc:2932] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
W20240501 19:56:45.701611 24924 consensus_peers.cc:485] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66215de525314ea180c164667f4e9f3f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.702483 24857 raft_consensus.cc:2932] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:45.703393 24725 raft_consensus.cc:2932] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:45.703797 24450 catalog_manager.cc:5484] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:45.725093 24727 consensus_queue.cc:237] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:45.726524 24857 raft_consensus.cc:1261] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.726887 24594 raft_consensus.cc:1261] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.726950 26678 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.727378 26678 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.729363 26677 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:45.729772 24594 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
W20240501 19:56:45.731523 24660 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb0bb0d3396a436c878f56d157f30eda. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.732204 24857 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:45.732846 24452 catalog_manager.cc:5484] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:45.754325 24857 consensus_queue.cc:237] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:45.755986 24594 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.756237 24986 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.756395 27731 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.756665 26944 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:45.759662 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d2603a318c34256a9fa28979ae41fec. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.764304 26944 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:45.764797 24986 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:45.768378 24446 catalog_manager.cc:5484] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:45.771072 24594 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:45.779794 24450 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 19:56:45.779920 24450 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 19:56:45.781574 24446 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:45.781759 27639 consensus_queue.cc:1035] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.783315 24450 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet ceed9cadcaa947b995ebc07e0e05fae9
I20240501 19:56:45.783638 24450 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet afc342897aa343fc930be1d27a2bb641
I20240501 19:56:45.783866 24450 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 382da83035804db1bc8743d1916bc376
I20240501 19:56:45.784900 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet afc342897aa343fc930be1d27a2bb641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:45.784901 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet 382da83035804db1bc8743d1916bc376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:45.784962 24969 tablet_service.cc:1514] Processing DeleteTablet for tablet ceed9cadcaa947b995ebc07e0e05fae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:45.785073 27626 tablet_replica.cc:330] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:45.785077 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet 382da83035804db1bc8743d1916bc376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:45.785120 24967 tablet_service.cc:1514] Processing DeleteTablet for tablet afc342897aa343fc930be1d27a2bb641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:45.785216 27626 raft_consensus.cc:2229] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.785204 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet ceed9cadcaa947b995ebc07e0e05fae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:45.785351 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet ceed9cadcaa947b995ebc07e0e05fae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:45.785341 27626 raft_consensus.cc:2258] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.785538 27624 tablet_replica.cc:330] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:45.785665 27624 raft_consensus.cc:2229] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.785797 27624 raft_consensus.cc:2258] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.785877 27626 ts_tablet_manager.cc:1875] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.785912 27625 tablet_replica.cc:330] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:45.786055 27625 raft_consensus.cc:2229] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.786237 27625 raft_consensus.cc:2258] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.786314 27624 ts_tablet_manager.cc:1875] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.786736 27625 ts_tablet_manager.cc:1875] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.787868 27626 ts_tablet_manager.cc:1888] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.787945 27626 log.cc:1196] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/382da83035804db1bc8743d1916bc376
I20240501 19:56:45.788179 27626 ts_tablet_manager.cc:1909] T 382da83035804db1bc8743d1916bc376 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:45.788604 27624 ts_tablet_manager.cc:1888] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.788681 27624 log.cc:1196] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/afc342897aa343fc930be1d27a2bb641
I20240501 19:56:45.788933 27625 ts_tablet_manager.cc:1888] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.788954 27624 ts_tablet_manager.cc:1909] T afc342897aa343fc930be1d27a2bb641 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:45.789012 27625 log.cc:1196] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/ceed9cadcaa947b995ebc07e0e05fae9
I20240501 19:56:45.789266 27625 ts_tablet_manager.cc:1909] T ceed9cadcaa947b995ebc07e0e05fae9 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:45.789469 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet afc342897aa343fc930be1d27a2bb641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:45.789688 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet afc342897aa343fc930be1d27a2bb641 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.789852 27623 tablet_replica.cc:330] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:45.789974 27625 tablet_replica.cc:330] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:45.789994 27623 raft_consensus.cc:2229] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.790107 27625 raft_consensus.cc:2229] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.790177 27623 raft_consensus.cc:2258] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.790278 27625 raft_consensus.cc:2258] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.790693 27623 ts_tablet_manager.cc:1875] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.790735 24594 raft_consensus.cc:1261] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.790820 27162 raft_consensus.cc:3037] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.791144 26607 consensus_queue.cc:1035] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.791260 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet ceed9cadcaa947b995ebc07e0e05fae9 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.789608 27624 tablet_replica.cc:330] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:45.791594 27624 raft_consensus.cc:2229] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.791714 27624 raft_consensus.cc:2258] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.791925 27162 raft_consensus.cc:1261] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.792202 27624 ts_tablet_manager.cc:1875] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.792809 27623 ts_tablet_manager.cc:1888] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.792867 27623 log.cc:1196] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/afc342897aa343fc930be1d27a2bb641
I20240501 19:56:45.792919 26607 consensus_queue.cc:1035] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.793116 27623 ts_tablet_manager.cc:1909] T afc342897aa343fc930be1d27a2bb641 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:45.793870 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet afc342897aa343fc930be1d27a2bb641 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.784901 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 382da83035804db1bc8743d1916bc376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:45.794420 27626 tablet_replica.cc:330] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:45.794548 27626 raft_consensus.cc:2229] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.794669 27626 raft_consensus.cc:2258] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.794858 24452 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet fb96a28495e549b1b4264eaa08611265 (table rebalancer_dynamic_table [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]). Expected version 1 got 0
I20240501 19:56:45.795955 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 382da83035804db1bc8743d1916bc376 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.796092 27626 ts_tablet_manager.cc:1875] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.796264 24446 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet fb96a28495e549b1b4264eaa08611265 (table rebalancer_dynamic_table [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]). Expected version 1 got 0
I20240501 19:56:45.797073 27625 ts_tablet_manager.cc:1875] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.797528 27639 consensus_queue.cc:1035] T 56017e72b6af48ccab39219fb7196bee P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.797806 24446 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet fb96a28495e549b1b4264eaa08611265 (table rebalancer_dynamic_table [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]). Expected version 1 got 0
I20240501 19:56:45.800062 27560 mvcc.cc:205] Tried to move back new op lower bound from 7022974590099607552 to 7022974589550137344. Current Snapshot: MvccSnapshot[applied={T|T < 7022974590099607552}]
I20240501 19:56:45.800269 27564 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:45.800460 27626 ts_tablet_manager.cc:1888] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.800536 27626 log.cc:1196] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/ceed9cadcaa947b995ebc07e0e05fae9
I20240501 19:56:45.800558 27564 tablet.cc:1620] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:45.800820 27626 ts_tablet_manager.cc:1909] T ceed9cadcaa947b995ebc07e0e05fae9 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:45.801949 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet ceed9cadcaa947b995ebc07e0e05fae9 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.802947 27624 ts_tablet_manager.cc:1888] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.803020 27624 log.cc:1196] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/ceed9cadcaa947b995ebc07e0e05fae9
I20240501 19:56:45.803298 27624 ts_tablet_manager.cc:1909] T ceed9cadcaa947b995ebc07e0e05fae9 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:45.803813 27624 tablet_replica.cc:330] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:45.803946 27624 raft_consensus.cc:2229] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.804064 27624 raft_consensus.cc:2258] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.804486 27624 ts_tablet_manager.cc:1875] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.804878 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet ceed9cadcaa947b995ebc07e0e05fae9 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.804899 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=f9ddf3e25ffb4b06a43ef41caf9a4f15] alter complete (version 1)
I20240501 19:56:45.805079 27625 ts_tablet_manager.cc:1888] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.805205 27625 log.cc:1196] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/382da83035804db1bc8743d1916bc376
I20240501 19:56:45.805616 27625 ts_tablet_manager.cc:1909] T 382da83035804db1bc8743d1916bc376 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:45.806166 27625 tablet_replica.cc:330] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:45.806329 27625 raft_consensus.cc:2229] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.806533 27625 raft_consensus.cc:2258] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.806677 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 382da83035804db1bc8743d1916bc376 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.806768 27624 ts_tablet_manager.cc:1888] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.806840 27624 log.cc:1196] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/382da83035804db1bc8743d1916bc376
I20240501 19:56:45.807126 27624 ts_tablet_manager.cc:1909] T 382da83035804db1bc8743d1916bc376 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:45.807180 27625 ts_tablet_manager.cc:1875] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.807845 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 382da83035804db1bc8743d1916bc376 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.808907 27625 ts_tablet_manager.cc:1888] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:45.808967 27625 log.cc:1196] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/afc342897aa343fc930be1d27a2bb641
I20240501 19:56:45.809201 27625 ts_tablet_manager.cc:1909] T afc342897aa343fc930be1d27a2bb641 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:45.809463 27807 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:45.810088 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet afc342897aa343fc930be1d27a2bb641 (table rebalance_test_table_2 [id=ecee71e04eb7403e978433da58c86ba2]) successfully deleted
I20240501 19:56:45.810210 27807 tablet.cc:1620] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:56:45.810981 27813 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:45.811844 27813 tablet.cc:1620] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:45.823369 24727 consensus_queue.cc:237] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:45.824712 24857 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.824800 24986 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.825266 26677 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.825667 26678 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:45.828075 24662 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 433347ce828b4d17a70308c421ab4f1f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.828704 26677 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:45.829964 24857 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:45.830410 24986 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:45.831869 24446 catalog_manager.cc:5484] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:45.851172 24727 consensus_queue.cc:237] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:45.852550 24594 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.852943 24857 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.853264 26678 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.853430 27637 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:45.856293 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b240b461b548a087fcabdeb938df1e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.856568 27637 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:45.858016 24594 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:45.858328 24857 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:45.860255 24452 catalog_manager.cc:5484] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:45.880070 24727 consensus_queue.cc:237] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:45.881505 24986 raft_consensus.cc:1261] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.881505 24594 raft_consensus.cc:1261] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.881914 26677 consensus_queue.cc:1035] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.882097 27840 consensus_queue.cc:1035] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:45.882318 24660 consensus_peers.cc:485] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d56fe9e75a9c418f817185542ae4c1d9. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.883913 26677 raft_consensus.cc:2932] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:45.884317 24986 raft_consensus.cc:2932] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:45.885830 24594 raft_consensus.cc:2932] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:45.887221 24450 catalog_manager.cc:5484] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:45.906268 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:45.912338 27564 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:45.912671 27564 tablet.cc:1620] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:45.912811 27813 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:45.913077 27813 tablet.cc:1620] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:45.913136 27807 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:45.913383 27807 tablet.cc:1620] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:56:45.916011 27356 consensus_queue.cc:1035] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.918622 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=f9ddf3e25ffb4b06a43ef41caf9a4f15] alter complete (version 2)
I20240501 19:56:45.920485 26885 consensus_queue.cc:1035] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.990134 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:45.994890 27267 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:45.996449 27406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:46.000897 24705 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:56:46.008724 27012 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:46.013425 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:46.019196 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:46.019338 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:46.022178 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb1e61bb542415caac5978943e6b3eb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.022544 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb1e61bb542415caac5978943e6b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.022807 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 022bb802aa6a4c66a71b7464492360ac (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.022854 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb1e61bb542415caac5978943e6b3eb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.023103 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022bb802aa6a4c66a71b7464492360ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.023134 27404 tablet_service.cc:1467] Processing CreateTablet for tablet 98850f18f10c4a57aaaf7a4bd9dce783 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.023290 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb1e61bb542415caac5978943e6b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.023766 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98850f18f10c4a57aaaf7a4bd9dce783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.025467 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb1e61bb542415caac5978943e6b3eb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.025770 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb1e61bb542415caac5978943e6b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.025807 27909 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:46.025938 27489 tablet_bootstrap.cc:492] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:46.026785 27909 tablet_bootstrap.cc:654] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.026916 27489 tablet_bootstrap.cc:654] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.028209 27910 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.028340 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 98850f18f10c4a57aaaf7a4bd9dce783 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.028429 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 98850f18f10c4a57aaaf7a4bd9dce783 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.028566 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98850f18f10c4a57aaaf7a4bd9dce783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.028717 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98850f18f10c4a57aaaf7a4bd9dce783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.029228 27910 tablet_bootstrap.cc:654] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.022854 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 022bb802aa6a4c66a71b7464492360ac (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.032156 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022bb802aa6a4c66a71b7464492360ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.034946 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 022bb802aa6a4c66a71b7464492360ac (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.035254 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022bb802aa6a4c66a71b7464492360ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.036891 27909 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:46.037039 27909 ts_tablet_manager.cc:1367] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:46.037415 27708 tablet_bootstrap.cc:492] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:46.037719 27909 raft_consensus.cc:348] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.037827 27909 raft_consensus.cc:374] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.037853 27909 raft_consensus.cc:729] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.037963 27909 consensus_queue.cc:260] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.038267 27708 tablet_bootstrap.cc:654] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.039098 27909 ts_tablet_manager.cc:1398] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.040905 27913 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:46.041680 27489 tablet_bootstrap.cc:492] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:46.041791 27489 ts_tablet_manager.cc:1367] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240501 19:56:46.042085 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:56:46.042320 27489 raft_consensus.cc:348] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.042454 27489 raft_consensus.cc:374] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.042490 27489 raft_consensus.cc:729] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.042661 27489 consensus_queue.cc:260] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.043574 27489 ts_tablet_manager.cc:1398] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:46.044528 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:46.052325 27807 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:46.053359 27913 tablet_bootstrap.cc:654] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.053709 27813 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:46.055460 27913 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:46.055430 27564 tablet.cc:1722] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:46.055545 27913 ts_tablet_manager.cc:1367] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:56:46.056097 27913 raft_consensus.cc:348] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.056233 27913 raft_consensus.cc:374] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.056280 27913 raft_consensus.cc:729] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.056429 27913 consensus_queue.cc:260] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.057222 27913 ts_tablet_manager.cc:1398] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.057436 27913 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:46.058532 27913 tablet_bootstrap.cc:654] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.062718 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=f9ddf3e25ffb4b06a43ef41caf9a4f15] alter complete (version 3)
I20240501 19:56:46.067144 27915 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:46.068959 27915 tablet_bootstrap.cc:654] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.069694 27913 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:46.069816 27913 ts_tablet_manager.cc:1367] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:46.069885 27708 tablet_bootstrap.cc:492] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:46.069967 27708 ts_tablet_manager.cc:1367] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.001s
I20240501 19:56:46.070300 27913 raft_consensus.cc:348] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.070412 27913 raft_consensus.cc:374] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.070449 27913 raft_consensus.cc:729] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.070461 27708 raft_consensus.cc:348] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.070590 27708 raft_consensus.cc:374] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.070637 27708 raft_consensus.cc:729] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.070749 27708 consensus_queue.cc:260] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.070778 27913 consensus_queue.cc:260] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.071022 27708 ts_tablet_manager.cc:1398] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:46.071630 27913 ts_tablet_manager.cc:1398] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.074425 27915 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:46.074528 27915 ts_tablet_manager.cc:1367] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:46.074931 27915 raft_consensus.cc:348] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.075062 27915 raft_consensus.cc:374] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.075135 27915 raft_consensus.cc:729] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.075275 27915 consensus_queue.cc:260] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.075584 27915 ts_tablet_manager.cc:1398] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:46.076503 27910 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:46.076611 27910 ts_tablet_manager.cc:1367] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.000s
I20240501 19:56:46.077000 27910 raft_consensus.cc:348] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.077096 27910 raft_consensus.cc:374] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.077126 27910 raft_consensus.cc:729] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.077251 27910 consensus_queue.cc:260] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.077596 27910 ts_tablet_manager.cc:1398] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.078864 27910 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.080755 27910 tablet_bootstrap.cc:654] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.081506 27637 raft_consensus.cc:482] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.081612 27637 raft_consensus.cc:504] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.081997 27637 leader_election.cc:288] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:46.082383 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98850f18f10c4a57aaaf7a4bd9dce783" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:46.082497 27031 raft_consensus.cc:2454] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:46.082787 24660 leader_election.cc:302] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:46.082934 27637 raft_consensus.cc:2781] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.082978 27637 raft_consensus.cc:482] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.083017 27637 raft_consensus.cc:3037] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.084028 27637 raft_consensus.cc:504] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.084321 27637 leader_election.cc:288] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:46.084599 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98850f18f10c4a57aaaf7a4bd9dce783" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:46.084720 27031 raft_consensus.cc:3037] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.085589 27031 raft_consensus.cc:2454] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:46.085868 24660 leader_election.cc:302] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:46.086035 27637 raft_consensus.cc:2781] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.086117 27637 raft_consensus.cc:686] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:46.082604 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98850f18f10c4a57aaaf7a4bd9dce783" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:46.086244 27637 consensus_queue.cc:237] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.084748 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98850f18f10c4a57aaaf7a4bd9dce783" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
W20240501 19:56:46.087086 24662 leader_election.cc:341] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:56:46.087244 24662 leader_election.cc:341] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:46.088335 24448 catalog_manager.cc:5484] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.091724 27910 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:46.091876 27910 ts_tablet_manager.cc:1367] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:56:46.092865 27910 raft_consensus.cc:348] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.092968 27910 raft_consensus.cc:374] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.093003 27910 raft_consensus.cc:729] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.093158 27910 consensus_queue.cc:260] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.093139 27677 consensus_queue.cc:1035] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.093407 27910 ts_tablet_manager.cc:1398] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.094322 27910 tablet_bootstrap.cc:492] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.095382 27910 tablet_bootstrap.cc:654] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.096179 27677 consensus_queue.cc:1035] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.097223 27910 tablet_bootstrap.cc:492] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:46.097312 27910 ts_tablet_manager.cc:1367] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:46.098105 27910 raft_consensus.cc:348] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.098256 27910 raft_consensus.cc:374] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.098335 27910 raft_consensus.cc:729] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.098517 27910 consensus_queue.cc:260] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.098853 27910 ts_tablet_manager.cc:1398] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:46.135622 27920 raft_consensus.cc:482] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.135587 24594 consensus_queue.cc:237] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.135725 27920 raft_consensus.cc:504] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.136153 27920 leader_election.cc:288] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:46.136473 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98850f18f10c4a57aaaf7a4bd9dce783" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:46.136646 27031 raft_consensus.cc:2379] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a6e54e86a9949e5970b42e8e32e7561 in current term 1: Already voted for candidate 9081292597be4a72a42b216fddcd7971 in this term.
I20240501 19:56:46.137348 24727 raft_consensus.cc:1261] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.137542 24986 raft_consensus.cc:1261] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.140415 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98850f18f10c4a57aaaf7a4bd9dce783" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:46.140770 27359 leader_election.cc:302] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971
I20240501 19:56:46.140980 27920 raft_consensus.cc:3037] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.141993 27920 raft_consensus.cc:2726] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:46.143154 27356 consensus_queue.cc:1035] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.143517 27356 consensus_queue.cc:1035] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:46.144170 24531 consensus_peers.cc:485] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d91e1c792c749e7883bb996d58d99ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.147806 26885 raft_consensus.cc:2932] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.148298 24727 raft_consensus.cc:2932] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.149506 26607 raft_consensus.cc:482] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.149590 26607 raft_consensus.cc:504] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.150082 26607 leader_election.cc:288] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:46.151167 24448 catalog_manager.cc:5484] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.151376 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022bb802aa6a4c66a71b7464492360ac" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:46.151517 27162 raft_consensus.cc:2454] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:46.151839 24924 leader_election.cc:302] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:46.153168 24986 raft_consensus.cc:2932] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.153731 27754 raft_consensus.cc:2781] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.153785 27754 raft_consensus.cc:482] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.153823 27754 raft_consensus.cc:3037] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.154685 27754 raft_consensus.cc:504] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.155048 27754 leader_election.cc:288] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:46.155856 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022bb802aa6a4c66a71b7464492360ac" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:46.156028 27424 raft_consensus.cc:2454] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:46.156251 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022bb802aa6a4c66a71b7464492360ac" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:46.156338 27425 raft_consensus.cc:3037] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.157325 27425 raft_consensus.cc:2454] T 022bb802aa6a4c66a71b7464492360ac P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:46.157747 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022bb802aa6a4c66a71b7464492360ac" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:46.157861 27162 raft_consensus.cc:3037] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.158809 27162 raft_consensus.cc:2454] T 022bb802aa6a4c66a71b7464492360ac P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:46.158929 24924 leader_election.cc:302] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:46.159214 26607 raft_consensus.cc:2781] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.159308 26607 raft_consensus.cc:686] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:46.159456 26607 consensus_queue.cc:237] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.162122 24445 catalog_manager.cc:5484] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:46.166154 24445 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:46.166272 24445 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:46.168267 24445 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet fb96a28495e549b1b4264eaa08611265
I20240501 19:56:46.168942 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet fb96a28495e549b1b4264eaa08611265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:46.169144 27625 tablet_replica.cc:330] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:46.169287 27625 raft_consensus.cc:2229] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.169361 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet fb96a28495e549b1b4264eaa08611265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:46.169467 27625 raft_consensus.cc:2258] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.169514 27626 tablet_replica.cc:330] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:46.169634 27626 raft_consensus.cc:2229] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.169749 27626 raft_consensus.cc:2258] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.170117 27625 ts_tablet_manager.cc:1875] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.170328 27626 ts_tablet_manager.cc:1875] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.171536 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet fb96a28495e549b1b4264eaa08611265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:46.172333 27625 ts_tablet_manager.cc:1888] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.172333 27626 ts_tablet_manager.cc:1888] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.172418 27626 log.cc:1196] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/fb96a28495e549b1b4264eaa08611265
I20240501 19:56:46.172479 27625 log.cc:1196] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/fb96a28495e549b1b4264eaa08611265
I20240501 19:56:46.172695 27626 ts_tablet_manager.cc:1909] T fb96a28495e549b1b4264eaa08611265 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:46.172703 27958 tablet_replica.cc:330] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:46.172734 27625 ts_tablet_manager.cc:1909] T fb96a28495e549b1b4264eaa08611265 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:46.172817 27958 raft_consensus.cc:2229] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.172922 27958 raft_consensus.cc:2258] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.173446 27958 ts_tablet_manager.cc:1875] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.173453 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet fb96a28495e549b1b4264eaa08611265 (table rebalancer_dynamic_table_new_name [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]) successfully deleted
I20240501 19:56:46.173969 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet fb96a28495e549b1b4264eaa08611265 (table rebalancer_dynamic_table_new_name [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]) successfully deleted
I20240501 19:56:46.175304 27959 ts_tablet_manager.cc:902] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:46.176005 27959 tablet_copy_client.cc:308] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:46.177217 27958 ts_tablet_manager.cc:1888] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.177279 27958 log.cc:1196] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/fb96a28495e549b1b4264eaa08611265
I20240501 19:56:46.177502 27958 ts_tablet_manager.cc:1909] T fb96a28495e549b1b4264eaa08611265 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:46.178539 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet fb96a28495e549b1b4264eaa08611265 (table rebalancer_dynamic_table_new_name [id=f9ddf3e25ffb4b06a43ef41caf9a4f15]) successfully deleted
I20240501 19:56:46.182821 24747 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet cb0bb0d3396a436c878f56d157f30eda from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:37455)
I20240501 19:56:46.182940 24747 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet cb0bb0d3396a436c878f56d157f30eda from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:37455: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-cb0bb0d3396a436c878f56d157f30eda
I20240501 19:56:46.184234 24747 tablet_copy_source_session.cc:205] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.185961 27959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0bb0d3396a436c878f56d157f30eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.188115 27959 tablet_copy_client.cc:783] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.188269 27959 tablet_copy_client.cc:647] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.189369 24594 consensus_queue.cc:237] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.190796 24986 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.190797 24857 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.191195 27356 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.191262 27959 tablet_copy_client.cc:523] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.192602 27959 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
W20240501 19:56:46.194344 24531 consensus_peers.cc:485] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 718c7f2f1f074e3c84534b502a4809a2. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.195561 27356 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.197479 27943 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.197858 24857 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.200888 24445 catalog_manager.cc:5484] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.201359 24986 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.204190 27926 raft_consensus.cc:482] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.204274 27926 raft_consensus.cc:504] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.204694 27926 leader_election.cc:288] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:46.209482 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb1e61bb542415caac5978943e6b3eb" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:46.209703 27278 raft_consensus.cc:2454] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:46.210027 26967 leader_election.cc:302] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:46.210285 27926 raft_consensus.cc:2781] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.210343 27926 raft_consensus.cc:482] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.210397 27926 raft_consensus.cc:3037] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.210455 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb1e61bb542415caac5978943e6b3eb" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:46.210611 27425 raft_consensus.cc:2454] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:46.211350 27926 raft_consensus.cc:504] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.211700 27926 leader_election.cc:288] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:46.212047 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb1e61bb542415caac5978943e6b3eb" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:46.212085 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb1e61bb542415caac5978943e6b3eb" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:46.212141 27278 raft_consensus.cc:3037] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.212181 27425 raft_consensus.cc:3037] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.213069 27425 raft_consensus.cc:2454] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:46.213150 27278 raft_consensus.cc:2454] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:46.213338 26967 leader_election.cc:302] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:46.213483 27926 raft_consensus.cc:2781] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.213543 27926 raft_consensus.cc:686] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:46.214366 27926 consensus_queue.cc:237] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.215994 24448 catalog_manager.cc:5484] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.227754 27959 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.228152 27959 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:56:46.228346 27959 ts_tablet_manager.cc:1367] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.036s user 0.032s sys 0.000s
I20240501 19:56:46.228790 27959 raft_consensus.cc:348] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:46.228919 27959 raft_consensus.cc:729] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.229058 27959 consensus_queue.cc:260] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:46.229393 27959 ts_tablet_manager.cc:1398] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:46.230213 24747 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-cb0bb0d3396a436c878f56d157f30eda received from {username='slave'} at 127.17.65.5:37455
I20240501 19:56:46.230305 24747 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-cb0bb0d3396a436c878f56d157f30eda on tablet cb0bb0d3396a436c878f56d157f30eda with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:46.231812 24727 consensus_queue.cc:237] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.233261 24594 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:46.233464 24662 consensus_peers.cc:485] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcfa6701a958416ca481e0349071caa0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.233357 27994 ts_tablet_manager.cc:902] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:46.233652 27637 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.233881 24857 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.234287 27637 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.234591 27994 tablet_copy_client.cc:308] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:56:46.236157 27840 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.236793 24857 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.238005 24594 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.239281 24448 catalog_manager.cc:5484] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.241235 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet 66215de525314ea180c164667f4e9f3f from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:37883)
I20240501 19:56:46.241369 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet 66215de525314ea180c164667f4e9f3f from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:37883: session id = 53e34bb09d8e41129a5683079ff63610-66215de525314ea180c164667f4e9f3f
I20240501 19:56:46.242722 25009 tablet_copy_source_session.cc:205] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.244491 27994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66215de525314ea180c164667f4e9f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.246814 27994 tablet_copy_client.cc:783] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.246970 27994 tablet_copy_client.cc:647] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.249688 27994 tablet_copy_client.cc:523] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.252019 27994 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:46.259733 28013 ts_tablet_manager.cc:902] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:46.260624 28013 tablet_copy_client.cc:308] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:56:46.262961 28023 ts_tablet_manager.cc:902] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:46.263715 28023 tablet_copy_client.cc:308] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:56:46.264907 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:39423)
I20240501 19:56:46.265017 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:39423: session id = 4a6e54e86a9949e5970b42e8e32e7561-3f2d49dcdd8f49d48d8a9bd6d10c332c
I20240501 19:56:46.266273 24878 tablet_copy_source_session.cc:205] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.267764 28013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.269068 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 6d2603a318c34256a9fa28979ae41fec from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:40849)
I20240501 19:56:46.269160 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 6d2603a318c34256a9fa28979ae41fec from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:40849: session id = 29d1a658e2b946e0b412e5fef82db9d9-6d2603a318c34256a9fa28979ae41fec
I20240501 19:56:46.270002 24878 tablet_copy_source_session.cc:205] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.270017 28013 tablet_copy_client.cc:783] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.270195 28013 tablet_copy_client.cc:647] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.270416 28029 ts_tablet_manager.cc:902] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:46.270697 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:46.270923 28029 tablet_copy_client.cc:308] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:46.271390 24747 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet 433347ce828b4d17a70308c421ab4f1f from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:55523)
I20240501 19:56:46.271489 24747 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet 433347ce828b4d17a70308c421ab4f1f from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:55523: session id = 4a6e54e86a9949e5970b42e8e32e7561-433347ce828b4d17a70308c421ab4f1f
I20240501 19:56:46.271955 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2603a318c34256a9fa28979ae41fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.272341 24747 tablet_copy_source_session.cc:205] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.273093 28029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433347ce828b4d17a70308c421ab4f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.274119 28023 tablet_copy_client.cc:783] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.274279 28023 tablet_copy_client.cc:647] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.275552 28029 tablet_copy_client.cc:783] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.275724 28029 tablet_copy_client.cc:647] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.277911 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3]), partition=
I20240501 19:56:46.278153 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.278357 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3]), partition=
I20240501 19:56:46.278576 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.280359 28023 tablet_copy_client.cc:523] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.280424 27915 tablet_bootstrap.cc:492] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:46.280836 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3]), partition=
I20240501 19:56:46.281033 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.281352 27915 tablet_bootstrap.cc:654] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.282932 27489 tablet_bootstrap.cc:492] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:46.283010 27909 tablet_bootstrap.cc:492] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:46.283069 28023 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:46.283833 27489 tablet_bootstrap.cc:654] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.283891 27909 tablet_bootstrap.cc:654] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.284863 28013 tablet_copy_client.cc:523] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.285182 24987 consensus_queue.cc:237] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:46.285894 27994 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.286226 27994 tablet_bootstrap.cc:492] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:46.286671 27994 ts_tablet_manager.cc:1367] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.002s
I20240501 19:56:46.287147 27994 raft_consensus.cc:348] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.287485 24725 raft_consensus.cc:1261] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.287536 24857 raft_consensus.cc:1261] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.287858 27489 tablet_bootstrap.cc:492] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:46.287930 27489 ts_tablet_manager.cc:1367] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:46.288323 27489 raft_consensus.cc:348] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.288372 26607 consensus_queue.cc:1035] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.288436 27489 raft_consensus.cc:374] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.288483 27489 raft_consensus.cc:729] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.288596 27489 consensus_queue.cc:260] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.288836 27489 ts_tablet_manager.cc:1398] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.289095 27994 raft_consensus.cc:729] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.289114 28013 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.289283 27994 consensus_queue.cc:260] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.290197 27994 ts_tablet_manager.cc:1398] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:46.290670 26607 consensus_queue.cc:1035] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.290767 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-66215de525314ea180c164667f4e9f3f received from {username='slave'} at 127.17.65.7:37883
I20240501 19:56:46.290831 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-66215de525314ea180c164667f4e9f3f on tablet 66215de525314ea180c164667f4e9f3f with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:46.291729 24451 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 19:56:46.291848 24451 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 19:56:46.295218 27909 tablet_bootstrap.cc:492] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:46.295315 27909 ts_tablet_manager.cc:1367] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240501 19:56:46.295472 24451 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 0d1f471ee79f41289c961066603539cb
W20240501 19:56:46.295490 24922 consensus_peers.cc:485] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09d33aa5ebfd404ca218b069767d216b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.295598 28029 tablet_copy_client.cc:523] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.295732 27909 raft_consensus.cc:348] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.295847 27909 raft_consensus.cc:374] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.295872 24451 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 01905e6ce63e4a67bc1ac51f84a05b74
I20240501 19:56:46.295900 27909 raft_consensus.cc:729] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.296020 27909 consensus_queue.cc:260] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.296226 24451 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet d784f58f5a314629a1f8449f11962fb6
I20240501 19:56:46.296296 27909 ts_tablet_manager.cc:1398] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.296545 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 01905e6ce63e4a67bc1ac51f84a05b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:46.296720 27625 tablet_replica.cc:330] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:46.296854 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet d784f58f5a314629a1f8449f11962fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:46.296875 27625 raft_consensus.cc:2229] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.297005 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet d784f58f5a314629a1f8449f11962fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:46.297065 27625 raft_consensus.cc:2258] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.297546 27625 ts_tablet_manager.cc:1875] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.298519 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 01905e6ce63e4a67bc1ac51f84a05b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:46.298563 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet d784f58f5a314629a1f8449f11962fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:46.298607 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet 01905e6ce63e4a67bc1ac51f84a05b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:46.298776 27626 tablet_replica.cc:330] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:46.298879 27626 raft_consensus.cc:2229] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.298977 27626 raft_consensus.cc:2258] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.299117 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d1f471ee79f41289c961066603539cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:46.299350 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d1f471ee79f41289c961066603539cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:46.299448 27626 ts_tablet_manager.cc:1875] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.299515 27624 tablet_replica.cc:330] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:46.299649 27624 raft_consensus.cc:2229] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.299749 27624 raft_consensus.cc:2258] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.300146 27624 ts_tablet_manager.cc:1875] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.301080 27626 ts_tablet_manager.cc:1888] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.301129 27626 log.cc:1196] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/01905e6ce63e4a67bc1ac51f84a05b74
I20240501 19:56:46.301321 27626 ts_tablet_manager.cc:1909] T 01905e6ce63e4a67bc1ac51f84a05b74 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:46.301810 27624 ts_tablet_manager.cc:1888] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.301880 27624 log.cc:1196] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/01905e6ce63e4a67bc1ac51f84a05b74
I20240501 19:56:46.302140 27624 ts_tablet_manager.cc:1909] T 01905e6ce63e4a67bc1ac51f84a05b74 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:46.302017 27718 raft_consensus.cc:2932] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.302412 24727 raft_consensus.cc:2932] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.302757 27626 tablet_replica.cc:330] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:46.302929 27626 raft_consensus.cc:2229] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.303136 27626 raft_consensus.cc:2258] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.303246 27915 tablet_bootstrap.cc:492] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:46.303329 27915 ts_tablet_manager.cc:1367] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20240501 19:56:46.303815 27915 raft_consensus.cc:348] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.303920 27915 raft_consensus.cc:374] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.303956 27915 raft_consensus.cc:729] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.303990 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 01905e6ce63e4a67bc1ac51f84a05b74 (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.304056 27915 consensus_queue.cc:260] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.304278 27626 ts_tablet_manager.cc:1875] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.304365 27915 ts_tablet_manager.cc:1398] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.304473 24857 raft_consensus.cc:2932] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.305725 27994 ts_tablet_manager.cc:902] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:46.305752 24448 catalog_manager.cc:5484] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.306140 27994 tablet_copy_client.cc:308] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:46.306713 24747 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet a4b240b461b548a087fcabdeb938df1e from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:38719)
I20240501 19:56:46.307925 27626 ts_tablet_manager.cc:1888] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.307993 27626 log.cc:1196] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d1f471ee79f41289c961066603539cb
I20240501 19:56:46.308264 27626 ts_tablet_manager.cc:1909] T 0d1f471ee79f41289c961066603539cb P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:46.308940 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 0d1f471ee79f41289c961066603539cb (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.310518 27625 ts_tablet_manager.cc:1888] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.310623 27625 log.cc:1196] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/01905e6ce63e4a67bc1ac51f84a05b74
I20240501 19:56:46.311887 27625 ts_tablet_manager.cc:1909] T 01905e6ce63e4a67bc1ac51f84a05b74 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:46.314021 27625 tablet_replica.cc:330] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:46.314098 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 01905e6ce63e4a67bc1ac51f84a05b74 (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.314965 27625 raft_consensus.cc:2229] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.315224 27625 raft_consensus.cc:2258] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.316083 27625 ts_tablet_manager.cc:1875] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.317613 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d1f471ee79f41289c961066603539cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:46.320791 24747 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet a4b240b461b548a087fcabdeb938df1e from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:38719: session id = 53e34bb09d8e41129a5683079ff63610-a4b240b461b548a087fcabdeb938df1e
I20240501 19:56:46.321763 24747 tablet_copy_source_session.cc:205] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.322785 27994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b240b461b548a087fcabdeb938df1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.325083 27994 tablet_copy_client.cc:783] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.325224 27994 tablet_copy_client.cc:647] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.325871 27624 tablet_replica.cc:330] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:46.325974 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 01905e6ce63e4a67bc1ac51f84a05b74 (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.326157 27624 raft_consensus.cc:2229] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.326345 27624 raft_consensus.cc:2258] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.327018 27624 ts_tablet_manager.cc:1875] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.329442 27994 tablet_copy_client.cc:523] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.330523 27624 ts_tablet_manager.cc:1888] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.330639 27624 log.cc:1196] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/d784f58f5a314629a1f8449f11962fb6
I20240501 19:56:46.330873 27994 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:46.330997 27624 ts_tablet_manager.cc:1909] T d784f58f5a314629a1f8449f11962fb6 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:46.332278 27624 tablet_replica.cc:330] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:46.332360 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet d784f58f5a314629a1f8449f11962fb6 (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.332551 27624 raft_consensus.cc:2229] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.332741 27624 raft_consensus.cc:2258] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.333293 27624 ts_tablet_manager.cc:1875] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.335350 27624 ts_tablet_manager.cc:1888] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.335721 27625 ts_tablet_manager.cc:1888] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.335815 27625 log.cc:1196] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/d784f58f5a314629a1f8449f11962fb6
I20240501 19:56:46.336088 27625 ts_tablet_manager.cc:1909] T d784f58f5a314629a1f8449f11962fb6 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:46.336915 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet d784f58f5a314629a1f8449f11962fb6 (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.337939 28029 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.339852 28023 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.340155 28023 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:56:46.340267 27624 log.cc:1196] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d1f471ee79f41289c961066603539cb
I20240501 19:56:46.340324 28023 ts_tablet_manager.cc:1367] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.057s user 0.028s sys 0.000s
I20240501 19:56:46.340704 27624 ts_tablet_manager.cc:1909] T 0d1f471ee79f41289c961066603539cb P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:46.340821 28023 raft_consensus.cc:348] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.340950 28023 raft_consensus.cc:729] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.341109 28023 consensus_queue.cc:260] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.342993 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 0d1f471ee79f41289c961066603539cb (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.343202 28023 ts_tablet_manager.cc:1398] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:46.345342 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-6d2603a318c34256a9fa28979ae41fec received from {username='slave'} at 127.17.65.6:40849
I20240501 19:56:46.345448 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-6d2603a318c34256a9fa28979ae41fec on tablet 6d2603a318c34256a9fa28979ae41fec with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:56:46.350136 28013 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.350463 28013 tablet_bootstrap.cc:492] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:56:46.350754 28013 ts_tablet_manager.cc:1367] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.064s user 0.028s sys 0.006s
I20240501 19:56:46.351356 28013 raft_consensus.cc:348] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.351552 28013 raft_consensus.cc:729] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.351774 28013 consensus_queue.cc:260] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.353761 28013 ts_tablet_manager.cc:1398] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:46.353997 28042 tablet_replica.cc:330] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:46.354194 28042 raft_consensus.cc:2229] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.354383 28042 raft_consensus.cc:2258] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.355158 28042 ts_tablet_manager.cc:1875] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.357424 28042 ts_tablet_manager.cc:1888] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.357563 28042 log.cc:1196] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/d784f58f5a314629a1f8449f11962fb6
I20240501 19:56:46.357903 28042 ts_tablet_manager.cc:1909] T d784f58f5a314629a1f8449f11962fb6 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:46.358215 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-3f2d49dcdd8f49d48d8a9bd6d10c332c received from {username='slave'} at 127.17.65.8:39423
I20240501 19:56:46.358301 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-3f2d49dcdd8f49d48d8a9bd6d10c332c on tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:46.358837 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet d784f58f5a314629a1f8449f11962fb6 (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.359472 28042 tablet_replica.cc:330] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:46.359602 28042 raft_consensus.cc:2229] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.359805 28042 raft_consensus.cc:2258] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.360347 28042 ts_tablet_manager.cc:1875] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.362089 27959 ts_tablet_manager.cc:902] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:46.362550 27959 tablet_copy_client.cc:308] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:46.362752 28042 ts_tablet_manager.cc:1888] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.363049 28042 log.cc:1196] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/0d1f471ee79f41289c961066603539cb
I20240501 19:56:46.363024 24747 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet d56fe9e75a9c418f817185542ae4c1d9 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:37455)
I20240501 19:56:46.363162 24747 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet d56fe9e75a9c418f817185542ae4c1d9 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:37455: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-d56fe9e75a9c418f817185542ae4c1d9
I20240501 19:56:46.363418 28042 ts_tablet_manager.cc:1909] T 0d1f471ee79f41289c961066603539cb P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:46.364136 24747 tablet_copy_source_session.cc:205] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.364288 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 0d1f471ee79f41289c961066603539cb (table rebalance_test_table_4 [id=3b3115cd780e41c3947d6d65be71908c]) successfully deleted
I20240501 19:56:46.365060 27959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56fe9e75a9c418f817185542ae4c1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.365271 27495 raft_consensus.cc:482] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.365379 27495 raft_consensus.cc:504] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.366657 27495 leader_election.cc:288] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:46.367133 27994 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.367385 27994 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:46.367542 27994 ts_tablet_manager.cc:1367] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.000s
I20240501 19:56:46.367975 27994 raft_consensus.cc:348] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.368049 27959 tablet_copy_client.cc:783] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.368077 27994 raft_consensus.cc:729] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.368193 27959 tablet_copy_client.cc:647] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.368198 27994 consensus_queue.cc:260] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.368916 27994 ts_tablet_manager.cc:1398] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.369362 27959 tablet_copy_client.cc:523] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.370550 27959 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:46.379073 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a1c69d4c7646a6bbed8ae982c0ce8c" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:46.379144 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a1c69d4c7646a6bbed8ae982c0ce8c" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:46.379237 27278 raft_consensus.cc:2454] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:46.379253 24727 raft_consensus.cc:2454] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:46.379628 27096 leader_election.cc:302] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:46.379863 27495 raft_consensus.cc:2781] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.379917 27495 raft_consensus.cc:482] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.379953 27495 raft_consensus.cc:3037] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.380987 24857 consensus_queue.cc:237] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.381253 28029 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.381291 27495 raft_consensus.cc:504] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.381594 28029 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:56:46.381656 27495 leader_election.cc:288] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:46.381752 28029 ts_tablet_manager.cc:1367] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.044s user 0.024s sys 0.004s
I20240501 19:56:46.382206 28029 raft_consensus.cc:348] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.382226 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a1c69d4c7646a6bbed8ae982c0ce8c" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:46.382330 28029 raft_consensus.cc:729] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.382335 27278 raft_consensus.cc:3037] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.382625 28029 consensus_queue.cc:260] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.382884 28029 ts_tablet_manager.cc:1398] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.383015 24986 raft_consensus.cc:1261] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.383263 24727 raft_consensus.cc:1261] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.383499 27792 consensus_queue.cc:1035] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.383882 24747 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-a4b240b461b548a087fcabdeb938df1e received from {username='slave'} at 127.17.65.7:38719
I20240501 19:56:46.383956 24747 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-a4b240b461b548a087fcabdeb938df1e on tablet a4b240b461b548a087fcabdeb938df1e with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:46.384171 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-433347ce828b4d17a70308c421ab4f1f received from {username='slave'} at 127.17.65.8:55523
I20240501 19:56:46.384238 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-433347ce828b4d17a70308c421ab4f1f on tablet 433347ce828b4d17a70308c421ab4f1f with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:46.384604 28072 consensus_queue.cc:1035] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.385821 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a1c69d4c7646a6bbed8ae982c0ce8c" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:46.385919 24727 raft_consensus.cc:3037] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.386953 24727 raft_consensus.cc:2454] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
W20240501 19:56:46.387425 24793 consensus_peers.cc:485] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1cf6622eed045e7a864b9e397618ecd. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.387516 27278 raft_consensus.cc:2454] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:46.388048 27096 leader_election.cc:302] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:46.388307 27495 raft_consensus.cc:2781] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.388458 27495 raft_consensus.cc:686] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:46.388722 27495 consensus_queue.cc:237] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:46.390245 28072 raft_consensus.cc:2932] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.390959 24445 catalog_manager.cc:5484] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:46.391841 24725 raft_consensus.cc:2932] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.395053 24451 catalog_manager.cc:5484] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.398717 24986 raft_consensus.cc:2932] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.413054 27959 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.413697 27959 tablet_bootstrap.cc:492] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:56:46.414094 27959 ts_tablet_manager.cc:1367] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.044s user 0.026s sys 0.000s
I20240501 19:56:46.414626 27959 raft_consensus.cc:348] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:46.414814 27959 raft_consensus.cc:729] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.415019 27959 consensus_queue.cc:260] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:46.415997 27959 ts_tablet_manager.cc:1398] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:46.416666 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-d56fe9e75a9c418f817185542ae4c1d9 received from {username='slave'} at 127.17.65.5:37455
I20240501 19:56:46.416762 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-d56fe9e75a9c418f817185542ae4c1d9 on tablet d56fe9e75a9c418f817185542ae4c1d9 with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:46.424762 24594 consensus_queue.cc:237] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.426555 24987 raft_consensus.cc:1261] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.426581 24857 raft_consensus.cc:1261] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.427019 27975 consensus_queue.cc:1035] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.427279 26885 consensus_queue.cc:1035] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:46.427685 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71a4aa3f94bf4a3b8d84cae1e1765ac9. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.429687 27975 raft_consensus.cc:2932] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.430102 24856 raft_consensus.cc:2932] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.431804 24987 raft_consensus.cc:2932] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.434667 24451 catalog_manager.cc:5484] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.455611 24857 consensus_queue.cc:237] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.457754 24987 raft_consensus.cc:1261] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.458098 24594 raft_consensus.cc:1261] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.458525 28072 consensus_queue.cc:1035] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.460040 28072 consensus_queue.cc:1035] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.462116 27484 raft_consensus.cc:2932] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
W20240501 19:56:46.463188 24793 consensus_peers.cc:485] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1141ea148d74ed594effc258ad44dd1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.464535 24594 raft_consensus.cc:2932] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.466275 24448 catalog_manager.cc:5484] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.468534 24987 raft_consensus.cc:2932] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.494015 24594 consensus_queue.cc:237] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.495620 24725 raft_consensus.cc:1261] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.495858 24987 raft_consensus.cc:1261] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:46.496287 24531 consensus_peers.cc:485] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6dbe23d20b34a348938dcc3571020b7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.497015 26885 consensus_queue.cc:1035] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.497545 26885 consensus_queue.cc:1035] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.499816 27356 raft_consensus.cc:2932] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.500248 24987 raft_consensus.cc:2932] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.502104 24725 raft_consensus.cc:2932] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.503360 24445 catalog_manager.cc:5484] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.514341 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:46.519277 27278 raft_consensus.cc:1261] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:46.519784 24725 raft_consensus.cc:1261] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:46.519869 28075 consensus_queue.cc:1035] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.520274 28075 consensus_queue.cc:1035] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.523588 24448 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (table rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3]). Expected version 1 got 0
I20240501 19:56:46.524056 24451 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (table rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3]). Expected version 1 got 0
I20240501 19:56:46.524276 27564 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:46.524622 27564 tablet.cc:1620] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:46.526995 24451 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (table rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3]). Expected version 1 got 0
I20240501 19:56:46.527971 28148 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:46.528504 28144 mvcc.cc:205] Tried to move back new op lower bound from 7022974593097076736 to 7022974592571203584. Current Snapshot: MvccSnapshot[applied={T|T < 7022974593097076736}]
I20240501 19:56:46.528724 24445 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (table rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3]). Expected version 1 got 0
I20240501 19:56:46.528972 28148 tablet.cc:1620] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:46.528836 28147 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:46.529567 28147 tablet.cc:1620] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:46.530318 24451 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3] alter complete (version 1)
I20240501 19:56:46.535835 24594 consensus_queue.cc:237] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:46.537142 24725 raft_consensus.cc:1261] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.537294 24987 raft_consensus.cc:1261] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:46.537389 24529 consensus_peers.cc:485] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f1c61cca30c4d878d921c15158a5947. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.537706 26885 consensus_queue.cc:1035] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.537925 27975 consensus_queue.cc:1035] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.539680 27975 raft_consensus.cc:2932] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.540135 24987 raft_consensus.cc:2932] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.541616 24725 raft_consensus.cc:2932] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.543134 24445 catalog_manager.cc:5484] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.563706 24594 consensus_queue.cc:237] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.565322 24857 raft_consensus.cc:1261] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:46.565517 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f89e81297b66402ea09bdb415fb7f385. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.565810 27943 consensus_queue.cc:1035] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.565927 24725 raft_consensus.cc:1261] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.566320 27943 consensus_queue.cc:1035] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.568044 26885 raft_consensus.cc:2932] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.568492 24725 raft_consensus.cc:2932] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.571027 24448 catalog_manager.cc:5484] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.572448 24857 raft_consensus.cc:2932] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.574266 26678 consensus_queue.cc:1035] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.580574 27637 consensus_queue.cc:1035] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.596503 28043 consensus_queue.cc:1035] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.601147 27754 consensus_queue.cc:1035] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.617727 24857 consensus_queue.cc:237] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.619220 24594 raft_consensus.cc:1261] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.619514 24987 raft_consensus.cc:1261] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:46.619572 24793 consensus_peers.cc:485] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f8f81836b9944aa964a8236912c0183. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.619628 28072 consensus_queue.cc:1035] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.619953 28114 consensus_queue.cc:1035] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.621652 28072 raft_consensus.cc:2932] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.622067 24987 raft_consensus.cc:2932] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.623457 24594 raft_consensus.cc:2932] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.624974 24451 catalog_manager.cc:5484] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:46.633395 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:46.639093 27564 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:46.639451 27564 tablet.cc:1620] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:46.640025 27994 ts_tablet_manager.cc:902] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:46.639986 28147 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:46.640039 28148 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:46.640286 28147 tablet.cc:1620] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:46.640360 28148 tablet.cc:1620] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:46.640471 27994 tablet_copy_client.cc:308] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:46.641067 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet fcfa6701a958416ca481e0349071caa0 from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:38719)
I20240501 19:56:46.641157 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet fcfa6701a958416ca481e0349071caa0 from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:38719: session id = 53e34bb09d8e41129a5683079ff63610-fcfa6701a958416ca481e0349071caa0
I20240501 19:56:46.642150 24746 tablet_copy_source_session.cc:205] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.643119 27994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfa6701a958416ca481e0349071caa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.645152 27994 tablet_copy_client.cc:783] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.645287 27994 tablet_copy_client.cc:647] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.646452 27994 tablet_copy_client.cc:523] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.647660 27994 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:46.647954 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=1db70f67a265469ba9c876a058e8def3] alter complete (version 2)
I20240501 19:56:46.653309 24594 consensus_queue.cc:237] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
W20240501 19:56:46.655006 24529 consensus_peers.cc:485] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e996e96ce69348c09adc99dc7d9c7125. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.655318 24857 raft_consensus.cc:1261] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.655720 24987 raft_consensus.cc:1261] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:46.655742 27356 consensus_queue.cc:1035] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.656154 27356 consensus_queue.cc:1035] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.657693 27943 raft_consensus.cc:2932] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.658092 24987 raft_consensus.cc:2932] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.659374 24857 raft_consensus.cc:2932] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.661550 24445 catalog_manager.cc:5484] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:46.673609 27994 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.673915 27994 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:46.674096 27994 ts_tablet_manager.cc:1367] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.003s
I20240501 19:56:46.674539 27994 raft_consensus.cc:348] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.674643 27994 raft_consensus.cc:729] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.674763 27994 consensus_queue.cc:260] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.675046 27994 ts_tablet_manager.cc:1398] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:46.675797 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-fcfa6701a958416ca481e0349071caa0 received from {username='slave'} at 127.17.65.7:38719
I20240501 19:56:46.675886 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-fcfa6701a958416ca481e0349071caa0 on tablet fcfa6701a958416ca481e0349071caa0 with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:46.684381 27926 consensus_queue.cc:1035] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.687381 28234 consensus_queue.cc:1035] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.715940 27278 raft_consensus.cc:1203] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:46.745141 27031 raft_consensus.cc:1203] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:46.749061 24705 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:56:46.749703 28023 ts_tablet_manager.cc:902] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:46.750212 28023 tablet_copy_client.cc:308] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:46.750792 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:46.757092 28029 ts_tablet_manager.cc:902] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:46.757674 28029 tablet_copy_client.cc:308] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:46.758827 24452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:46.762707 27564 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:46.764083 27267 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:46.770114 28147 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:46.773710 27031 raft_consensus.cc:1203] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:46.776459 28148 tablet.cc:1722] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:46.780275 27011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:46.780699 24451 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=1db70f67a265469ba9c876a058e8def3] alter complete (version 3)
I20240501 19:56:46.782047 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:46.782311 27142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:46.788012 24574 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:46.788288 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 4851f59ff041424bae97a3ecaa9e191b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.788429 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed0e67f44543d09dbeda7100c7e0d6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.788571 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4851f59ff041424bae97a3ecaa9e191b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.788682 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed0e67f44543d09dbeda7100c7e0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.790897 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed0e67f44543d09dbeda7100c7e0d6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.791117 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed0e67f44543d09dbeda7100c7e0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.793288 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 718c7f2f1f074e3c84534b502a4809a2 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:56477)
I20240501 19:56:46.793434 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 718c7f2f1f074e3c84534b502a4809a2 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:56477: session id = 4a6e54e86a9949e5970b42e8e32e7561-718c7f2f1f074e3c84534b502a4809a2
I20240501 19:56:46.794737 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 92906e09f3474d7987082e0d33304886 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.794878 24616 tablet_copy_source_session.cc:205] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.794986 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92906e09f3474d7987082e0d33304886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.797148 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 4851f59ff041424bae97a3ecaa9e191b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.797379 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4851f59ff041424bae97a3ecaa9e191b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.799350 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 4851f59ff041424bae97a3ecaa9e191b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.799630 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4851f59ff041424bae97a3ecaa9e191b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.800171 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 92906e09f3474d7987082e0d33304886 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.800374 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92906e09f3474d7987082e0d33304886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.801177 24446 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 19:56:46.801313 24446 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 19:56:46.802047 28029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718c7f2f1f074e3c84534b502a4809a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.802721 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed0e67f44543d09dbeda7100c7e0d6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.802950 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed0e67f44543d09dbeda7100c7e0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.804078 28029 tablet_copy_client.cc:783] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.804209 28029 tablet_copy_client.cc:647] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.805107 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 92906e09f3474d7987082e0d33304886 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.805301 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92906e09f3474d7987082e0d33304886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.808072 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 0d91e1c792c749e7883bb996d58d99ea from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:53099)
I20240501 19:56:46.808190 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 0d91e1c792c749e7883bb996d58d99ea from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:53099: session id = 29d1a658e2b946e0b412e5fef82db9d9-0d91e1c792c749e7883bb996d58d99ea
I20240501 19:56:46.809152 24616 tablet_copy_source_session.cc:205] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.809599 28291 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:46.810480 28291 tablet_bootstrap.cc:654] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.810938 24446 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 9405d611c5a04984bb7cfb24f25d768d
I20240501 19:56:46.811254 24446 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 4 replicas of tablet a1cf6622eed045e7a864b9e397618ecd
I20240501 19:56:46.811518 24446 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 6c35e1be84514add973014e564ad845c
I20240501 19:56:46.811594 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d91e1c792c749e7883bb996d58d99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.812104 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cf6622eed045e7a864b9e397618ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:46.812268 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 9405d611c5a04984bb7cfb24f25d768d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:46.812325 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c35e1be84514add973014e564ad845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:46.812384 28042 tablet_replica.cc:330] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:46.812526 28042 raft_consensus.cc:2229] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.812784 28042 raft_consensus.cc:2258] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.813544 28023 tablet_copy_client.cc:783] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.813570 28042 ts_tablet_manager.cc:1875] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.813706 28023 tablet_copy_client.cc:647] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.814129 24967 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cf6622eed045e7a864b9e397618ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:46.814173 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 9405d611c5a04984bb7cfb24f25d768d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:46.814301 27625 tablet_replica.cc:330] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:46.814435 27625 raft_consensus.cc:2229] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.814635 27625 raft_consensus.cc:2258] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.815127 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c35e1be84514add973014e564ad845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:46.815325 27625 ts_tablet_manager.cc:1875] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.816049 28042 ts_tablet_manager.cc:1888] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.816112 28042 log.cc:1196] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/a1cf6622eed045e7a864b9e397618ecd
I20240501 19:56:46.816361 28042 ts_tablet_manager.cc:1909] T a1cf6622eed045e7a864b9e397618ecd P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:46.816553 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cf6622eed045e7a864b9e397618ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:46.816998 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c35e1be84514add973014e564ad845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:46.817328 27625 ts_tablet_manager.cc:1888] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.817389 27625 log.cc:1196] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/a1cf6622eed045e7a864b9e397618ecd
I20240501 19:56:46.817651 27625 ts_tablet_manager.cc:1909] T a1cf6622eed045e7a864b9e397618ecd P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:46.818182 27625 tablet_replica.cc:330] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:46.818300 27625 raft_consensus.cc:2229] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.818485 27625 raft_consensus.cc:2258] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.818612 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 9405d611c5a04984bb7cfb24f25d768d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:46.818727 28023 tablet_copy_client.cc:523] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.818992 27624 tablet_replica.cc:330] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:46.819154 27624 raft_consensus.cc:2229] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.819315 27624 raft_consensus.cc:2258] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.819940 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet a1cf6622eed045e7a864b9e397618ecd (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
I20240501 19:56:46.819999 28023 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:46.820464 27624 ts_tablet_manager.cc:1875] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.822458 27624 ts_tablet_manager.cc:1888] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.822513 27624 log.cc:1196] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/a1cf6622eed045e7a864b9e397618ecd
I20240501 19:56:46.822777 27624 ts_tablet_manager.cc:1909] T a1cf6622eed045e7a864b9e397618ecd P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:46.823249 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet a1cf6622eed045e7a864b9e397618ecd (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
I20240501 19:56:46.823415 28289 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:46.823709 27406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:46.824332 28289 tablet_bootstrap.cc:654] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.824640 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cf6622eed045e7a864b9e397618ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:46.825937 27624 tablet_replica.cc:330] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:46.826025 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet a1cf6622eed045e7a864b9e397618ecd (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
I20240501 19:56:46.826069 27624 raft_consensus.cc:2229] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.826261 27624 raft_consensus.cc:2258] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.826766 27624 ts_tablet_manager.cc:1875] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.828567 27624 ts_tablet_manager.cc:1888] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.828625 27624 log.cc:1196] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/6c35e1be84514add973014e564ad845c
I20240501 19:56:46.828882 27624 ts_tablet_manager.cc:1909] T 6c35e1be84514add973014e564ad845c P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:46.829002 28292 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:46.829775 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 6c35e1be84514add973014e564ad845c (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
I20240501 19:56:46.829970 27624 tablet_replica.cc:330] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:46.830133 27624 raft_consensus.cc:2229] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.830413 27624 raft_consensus.cc:2258] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.830991 27624 ts_tablet_manager.cc:1875] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.831452 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:46.833547 28293 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.824251 27625 ts_tablet_manager.cc:1875] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.836171 27425 raft_consensus.cc:1203] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:46.836961 28293 tablet_bootstrap.cc:654] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.837981 27624 ts_tablet_manager.cc:1888] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.837977 27425 raft_consensus.cc:1203] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:46.838053 27624 log.cc:1196] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/9405d611c5a04984bb7cfb24f25d768d
I20240501 19:56:46.838933 28292 tablet_bootstrap.cc:654] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.841718 27625 ts_tablet_manager.cc:1888] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.843386 27624 ts_tablet_manager.cc:1909] T 9405d611c5a04984bb7cfb24f25d768d P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:46.843626 27625 log.cc:1196] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/9405d611c5a04984bb7cfb24f25d768d
I20240501 19:56:46.843963 27625 ts_tablet_manager.cc:1909] T 9405d611c5a04984bb7cfb24f25d768d P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:46.844087 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 9405d611c5a04984bb7cfb24f25d768d (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
I20240501 19:56:46.848592 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 9405d611c5a04984bb7cfb24f25d768d (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
I20240501 19:56:46.848753 27625 tablet_replica.cc:330] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:46.848889 27625 raft_consensus.cc:2229] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.849025 27625 raft_consensus.cc:2258] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.849467 27625 ts_tablet_manager.cc:1875] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.851506 27625 ts_tablet_manager.cc:1888] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.851637 27625 log.cc:1196] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c35e1be84514add973014e564ad845c
I20240501 19:56:46.851969 27625 ts_tablet_manager.cc:1909] T 6c35e1be84514add973014e564ad845c P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:46.854600 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 6c35e1be84514add973014e564ad845c (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
W20240501 19:56:46.855489 24440 catalog_manager.cc:4797] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): delete failed for tablet a1cf6622eed045e7a864b9e397618ecd because the tablet was not found. No further retry: Not found: Tablet not found: a1cf6622eed045e7a864b9e397618ecd
I20240501 19:56:46.856377 28290 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:46.856695 28293 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:46.856777 28293 ts_tablet_manager.cc:1367] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 19:56:46.857256 28293 raft_consensus.cc:348] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.857370 28293 raft_consensus.cc:374] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.857409 28293 raft_consensus.cc:729] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.857537 28290 tablet_bootstrap.cc:654] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.857522 28293 consensus_queue.cc:260] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.857789 28293 ts_tablet_manager.cc:1398] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.857952 28293 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.858816 28293 tablet_bootstrap.cc:654] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.862004 27278 raft_consensus.cc:1203] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:46.863008 28292 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:46.863152 28292 ts_tablet_manager.cc:1367] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240501 19:56:46.863875 28292 raft_consensus.cc:348] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.863984 28292 raft_consensus.cc:374] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.864023 28292 raft_consensus.cc:729] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.864140 28292 consensus_queue.cc:260] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.864370 28292 ts_tablet_manager.cc:1398] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.864517 28292 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:46.865377 28292 tablet_bootstrap.cc:654] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.866104 28293 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:46.866178 28293 ts_tablet_manager.cc:1367] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:46.866427 27959 ts_tablet_manager.cc:902] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:46.866628 28293 raft_consensus.cc:348] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.866744 28293 raft_consensus.cc:374] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.866776 28293 raft_consensus.cc:729] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.866899 28293 consensus_queue.cc:260] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.866979 27959 tablet_copy_client.cc:308] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:56:46.868038 27162 raft_consensus.cc:1203] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:46.870720 28293 ts_tablet_manager.cc:1398] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:46.873543 28309 ts_tablet_manager.cc:902] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:46.874150 28309 tablet_copy_client.cc:308] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:46.874614 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet f6dbe23d20b34a348938dcc3571020b7 from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:53099)
I20240501 19:56:46.874732 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet f6dbe23d20b34a348938dcc3571020b7 from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:53099: session id = 29d1a658e2b946e0b412e5fef82db9d9-f6dbe23d20b34a348938dcc3571020b7
I20240501 19:56:46.875656 24616 tablet_copy_source_session.cc:205] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.875898 28289 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:46.875990 28289 ts_tablet_manager.cc:1367] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20240501 19:56:46.876451 28309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6dbe23d20b34a348938dcc3571020b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.876430 28289 raft_consensus.cc:348] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.876533 28289 raft_consensus.cc:374] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.876559 28289 raft_consensus.cc:729] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.876672 28289 consensus_queue.cc:260] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.876911 28289 ts_tablet_manager.cc:1398] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.877071 28289 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:46.877943 28289 tablet_bootstrap.cc:654] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.878463 28309 tablet_copy_client.cc:783] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.879350 28029 tablet_copy_client.cc:523] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.879458 28309 tablet_copy_client.cc:647] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.880688 28029 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.880928 28309 tablet_copy_client.cc:523] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.882170 27994 ts_tablet_manager.cc:902] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:46.882315 28309 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:46.882664 27994 tablet_copy_client.cc:308] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:56:46.883147 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet e1141ea148d74ed594effc258ad44dd1 from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:44335)
I20240501 19:56:46.883250 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet e1141ea148d74ed594effc258ad44dd1 from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:44335: session id = 53e34bb09d8e41129a5683079ff63610-e1141ea148d74ed594effc258ad44dd1
I20240501 19:56:46.884155 24878 tablet_copy_source_session.cc:205] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.885308 28290 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:46.885416 28292 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:46.885530 28292 ts_tablet_manager.cc:1367] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240501 19:56:46.885840 28290 ts_tablet_manager.cc:1367] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240501 19:56:46.886090 28292 raft_consensus.cc:348] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.886210 28292 raft_consensus.cc:374] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.886250 28292 raft_consensus.cc:729] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.886404 28292 consensus_queue.cc:260] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.886605 28290 raft_consensus.cc:348] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.886734 28292 ts_tablet_manager.cc:1398] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.886752 28290 raft_consensus.cc:374] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.886866 28290 raft_consensus.cc:729] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.887054 28290 consensus_queue.cc:260] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.887603 28290 ts_tablet_manager.cc:1398] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.889273 27994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1141ea148d74ed594effc258ad44dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.892067 27994 tablet_copy_client.cc:783] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.892200 27994 tablet_copy_client.cc:647] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.892557 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet 09d33aa5ebfd404ca218b069767d216b from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:52411)
I20240501 19:56:46.892786 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet 09d33aa5ebfd404ca218b069767d216b from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:52411: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-09d33aa5ebfd404ca218b069767d216b
I20240501 19:56:46.893563 27994 tablet_copy_client.cc:523] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.893700 25009 tablet_copy_source_session.cc:205] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.894539 27959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d33aa5ebfd404ca218b069767d216b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.894779 27994 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:46.896627 27959 tablet_copy_client.cc:783] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.896775 27959 tablet_copy_client.cc:647] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.897122 28289 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:46.897234 28289 ts_tablet_manager.cc:1367] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:56:46.897748 28289 raft_consensus.cc:348] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.897866 28289 raft_consensus.cc:374] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.897907 28289 raft_consensus.cc:729] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.898036 28289 consensus_queue.cc:260] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:46.898308 28289 ts_tablet_manager.cc:1398] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.900816 27959 tablet_copy_client.cc:523] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.902449 27959 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:46.903822 28295 tablet_replica.cc:330] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:46.904033 28295 raft_consensus.cc:2229] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.904075 28291 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:46.904170 28291 ts_tablet_manager.cc:1367] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.000s
I20240501 19:56:46.904237 28295 raft_consensus.cc:2258] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.904670 28291 raft_consensus.cc:348] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.904785 28291 raft_consensus.cc:374] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.904812 28291 raft_consensus.cc:729] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.904923 28291 consensus_queue.cc:260] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.905155 28295 ts_tablet_manager.cc:1875] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.905170 28291 ts_tablet_manager.cc:1398] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.905339 28291 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:46.906188 28291 tablet_bootstrap.cc:654] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.912775 27639 raft_consensus.cc:482] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.912871 27639 raft_consensus.cc:504] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.913316 27639 leader_election.cc:288] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:46.915745 28023 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.916119 28023 tablet_bootstrap.cc:492] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:56:46.916354 28023 ts_tablet_manager.cc:1367] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.096s user 0.024s sys 0.004s
I20240501 19:56:46.916875 28295 ts_tablet_manager.cc:1888] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.916918 28023 raft_consensus.cc:348] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.917068 28023 raft_consensus.cc:729] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.917014 28295 log.cc:1196] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/9405d611c5a04984bb7cfb24f25d768d
I20240501 19:56:46.917260 28023 consensus_queue.cc:260] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.917408 28295 ts_tablet_manager.cc:1909] T 9405d611c5a04984bb7cfb24f25d768d P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:46.918017 28295 tablet_replica.cc:330] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:46.918109 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 9405d611c5a04984bb7cfb24f25d768d (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
I20240501 19:56:46.918251 28295 raft_consensus.cc:2229] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.918422 28295 raft_consensus.cc:2258] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.919011 28295 ts_tablet_manager.cc:1875] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.919678 28291 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:46.919803 28291 ts_tablet_manager.cc:1367] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:46.920150 28327 ts_tablet_manager.cc:902] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:46.920293 28291 raft_consensus.cc:348] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.920401 28291 raft_consensus.cc:374] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.920439 28291 raft_consensus.cc:729] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.920555 28291 consensus_queue.cc:260] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.920830 28291 ts_tablet_manager.cc:1398] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.921501 24452 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:46.921623 24452 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:46.919672 28023 ts_tablet_manager.cc:1398] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:46.923821 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-0d91e1c792c749e7883bb996d58d99ea received from {username='slave'} at 127.17.65.6:53099
I20240501 19:56:46.923899 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-0d91e1c792c749e7883bb996d58d99ea on tablet 0d91e1c792c749e7883bb996d58d99ea with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:56:46.927281 24452 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 71a1c69d4c7646a6bbed8ae982c0ce8c
I20240501 19:56:46.927978 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:46.928069 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:56:46.928157 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a1c69d4c7646a6bbed8ae982c0ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:46.928303 28297 tablet_replica.cc:330] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:46.928427 28297 raft_consensus.cc:2229] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.928443 27624 tablet_replica.cc:330] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:46.928557 27624 raft_consensus.cc:2229] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.928660 27624 raft_consensus.cc:2258] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.928721 28297 raft_consensus.cc:2258] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.930351 27624 ts_tablet_manager.cc:1875] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.932613 27624 ts_tablet_manager.cc:1888] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.932682 27624 log.cc:1196] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/71a1c69d4c7646a6bbed8ae982c0ce8c
I20240501 19:56:46.932942 27624 ts_tablet_manager.cc:1909] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:46.933813 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (table rebalancer_dynamic_table_new_name [id=1db70f67a265469ba9c876a058e8def3]) successfully deleted
I20240501 19:56:46.934120 28297 ts_tablet_manager.cc:1875] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.934701 28309 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.935067 28309 tablet_bootstrap.cc:492] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:56:46.935276 28309 ts_tablet_manager.cc:1367] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.053s user 0.023s sys 0.005s
I20240501 19:56:46.935606 28295 ts_tablet_manager.cc:1888] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:46.935688 28295 log.cc:1196] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c35e1be84514add973014e564ad845c
I20240501 19:56:46.935864 28309 raft_consensus.cc:348] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.936018 28309 raft_consensus.cc:729] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.936013 28295 ts_tablet_manager.cc:1909] T 6c35e1be84514add973014e564ad845c P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:46.936277 28309 consensus_queue.cc:260] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.936702 28297 ts_tablet_manager.cc:1888] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.936687 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 6c35e1be84514add973014e564ad845c (table rebalance_test_table_6 [id=db353e9b5d3e45448f4f999f6d406d58]) successfully deleted
I20240501 19:56:46.936765 28297 log.cc:1196] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/71a1c69d4c7646a6bbed8ae982c0ce8c
I20240501 19:56:46.936952 28332 tablet_replica.cc:330] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:56:46.937026 28297 ts_tablet_manager.cc:1909] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:46.937266 28332 raft_consensus.cc:2229] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.937434 28332 raft_consensus.cc:2258] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.937712 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (table rebalancer_dynamic_table_new_name [id=1db70f67a265469ba9c876a058e8def3]) successfully deleted
I20240501 19:56:46.938338 28332 ts_tablet_manager.cc:1875] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.938959 28309 ts_tablet_manager.cc:1398] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:46.940292 28332 ts_tablet_manager.cc:1888] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:46.940372 28332 log.cc:1196] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/71a1c69d4c7646a6bbed8ae982c0ce8c
I20240501 19:56:46.940629 28332 ts_tablet_manager.cc:1909] T 71a1c69d4c7646a6bbed8ae982c0ce8c P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:56:46.941205 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-f6dbe23d20b34a348938dcc3571020b7 received from {username='slave'} at 127.17.65.6:53099
I20240501 19:56:46.941278 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-f6dbe23d20b34a348938dcc3571020b7 on tablet f6dbe23d20b34a348938dcc3571020b7 with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:56:46.943573 28075 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.943661 28075 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.944051 28075 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:46.945058 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92906e09f3474d7987082e0d33304886" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:46.945173 27278 raft_consensus.cc:2454] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:46.946410 27098 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:46.947115 28075 raft_consensus.cc:2781] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.947173 28075 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.947209 28075 raft_consensus.cc:3037] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.948186 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 71a1c69d4c7646a6bbed8ae982c0ce8c (table rebalancer_dynamic_table_new_name [id=1db70f67a265469ba9c876a058e8def3]) successfully deleted
I20240501 19:56:46.951390 28075 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.951745 28075 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:46.952971 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4851f59ff041424bae97a3ecaa9e191b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:46.953094 27031 raft_consensus.cc:2454] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:46.953401 27227 leader_election.cc:302] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:46.953567 27639 raft_consensus.cc:2781] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.953610 27639 raft_consensus.cc:482] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.953657 27639 raft_consensus.cc:3037] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.954571 27639 raft_consensus.cc:504] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.954917 27639 leader_election.cc:288] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:46.955596 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4851f59ff041424bae97a3ecaa9e191b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:46.955703 27031 raft_consensus.cc:3037] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.956614 27031 raft_consensus.cc:2454] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:46.956836 27959 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.956900 27994 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.957056 27227 leader_election.cc:302] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:46.957198 27959 tablet_bootstrap.cc:492] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:56:46.957211 27994 tablet_bootstrap.cc:492] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:46.957362 27959 ts_tablet_manager.cc:1367] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.055s user 0.022s sys 0.004s
I20240501 19:56:46.957365 27994 ts_tablet_manager.cc:1367] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.063s user 0.028s sys 0.000s
I20240501 19:56:46.957619 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92906e09f3474d7987082e0d33304886" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:46.957718 27278 raft_consensus.cc:3037] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.957801 27959 raft_consensus.cc:348] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:46.957808 27994 raft_consensus.cc:348] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.957888 27959 raft_consensus.cc:729] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.957896 27994 raft_consensus.cc:729] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.958003 27994 consensus_queue.cc:260] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:46.958020 27959 consensus_queue.cc:260] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:46.958328 27994 ts_tablet_manager.cc:1398] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.958550 27278 raft_consensus.cc:2454] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:46.958808 27959 ts_tablet_manager.cc:1398] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.958874 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-e1141ea148d74ed594effc258ad44dd1 received from {username='slave'} at 127.17.65.7:44335
I20240501 19:56:46.958947 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-e1141ea148d74ed594effc258ad44dd1 on tablet e1141ea148d74ed594effc258ad44dd1 with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:46.959151 28334 raft_consensus.cc:2781] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.959324 28334 raft_consensus.cc:686] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:46.959378 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-09d33aa5ebfd404ca218b069767d216b received from {username='slave'} at 127.17.65.5:52411
I20240501 19:56:46.959465 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-09d33aa5ebfd404ca218b069767d216b on tablet 09d33aa5ebfd404ca218b069767d216b with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:46.959453 28334 consensus_queue.cc:237] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:46.960255 27098 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:46.960439 28075 raft_consensus.cc:2781] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.960522 28075 raft_consensus.cc:686] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:46.960634 28075 consensus_queue.cc:237] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.962635 24448 catalog_manager.cc:5484] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:46.963802 24452 catalog_manager.cc:5484] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:46.969491 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92906e09f3474d7987082e0d33304886" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:46.969492 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92906e09f3474d7987082e0d33304886" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:46.969632 27424 raft_consensus.cc:3037] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.970589 27424 raft_consensus.cc:2454] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:46.974072 28327 tablet_copy_client.cc:308] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:46.974957 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:56477)
I20240501 19:56:46.975100 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:56477: session id = 4a6e54e86a9949e5970b42e8e32e7561-71a4aa3f94bf4a3b8d84cae1e1765ac9
I20240501 19:56:46.975420 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4851f59ff041424bae97a3ecaa9e191b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:46.975499 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4851f59ff041424bae97a3ecaa9e191b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:46.975579 27162 raft_consensus.cc:2454] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:46.976153 24616 tablet_copy_source_session.cc:205] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.976925 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.977008 28029 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:46.977255 28029 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:56:46.977422 28029 ts_tablet_manager.cc:1367] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.097s user 0.015s sys 0.008s
I20240501 19:56:46.977880 28029 raft_consensus.cc:348] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.978004 28029 raft_consensus.cc:729] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.978181 28029 consensus_queue.cc:260] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:46.978538 28029 ts_tablet_manager.cc:1398] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.979212 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-718c7f2f1f074e3c84534b502a4809a2 received from {username='slave'} at 127.17.65.8:56477
I20240501 19:56:46.979285 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-718c7f2f1f074e3c84534b502a4809a2 on tablet 718c7f2f1f074e3c84534b502a4809a2 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:46.979719 28327 tablet_copy_client.cc:783] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.979853 28327 tablet_copy_client.cc:647] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.981290 28327 tablet_copy_client.cc:523] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.982570 28327 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:46.987927 27959 ts_tablet_manager.cc:902] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:46.988338 27959 tablet_copy_client.cc:308] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:46.988740 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 6f1c61cca30c4d878d921c15158a5947 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:41295)
I20240501 19:56:46.988829 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 6f1c61cca30c4d878d921c15158a5947 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:41295: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-6f1c61cca30c4d878d921c15158a5947
I20240501 19:56:46.989688 24616 tablet_copy_source_session.cc:205] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:46.990590 27959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1c61cca30c4d878d921c15158a5947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.992738 27959 tablet_copy_client.cc:783] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:46.992872 27959 tablet_copy_client.cc:647] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:46.993969 27959 tablet_copy_client.cc:523] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:46.995224 27959 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:47.011386 28327 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:47.011777 28327 tablet_bootstrap.cc:492] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:56:47.012010 28327 ts_tablet_manager.cc:1367] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.029s user 0.024s sys 0.004s
I20240501 19:56:47.012445 28327 raft_consensus.cc:348] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:47.012571 28327 raft_consensus.cc:729] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.012761 28327 consensus_queue.cc:260] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:47.013132 28327 ts_tablet_manager.cc:1398] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.014183 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-71a4aa3f94bf4a3b8d84cae1e1765ac9 received from {username='slave'} at 127.17.65.8:56477
I20240501 19:56:47.014261 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-71a4aa3f94bf4a3b8d84cae1e1765ac9 on tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:47.023762 27959 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:47.024135 27959 tablet_bootstrap.cc:492] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:56:47.024353 27959 ts_tablet_manager.cc:1367] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.003s
I20240501 19:56:47.024869 27959 raft_consensus.cc:348] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.024986 27959 raft_consensus.cc:729] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.025213 27959 consensus_queue.cc:260] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.025498 27959 ts_tablet_manager.cc:1398] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.026114 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-6f1c61cca30c4d878d921c15158a5947 received from {username='slave'} at 127.17.65.5:41295
I20240501 19:56:47.026194 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-6f1c61cca30c4d878d921c15158a5947 on tablet 6f1c61cca30c4d878d921c15158a5947 with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:47.028795 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:47.036298 27011 tablet_service.cc:1467] Processing CreateTablet for tablet b930ffbf00094a80b6b03b79a1b1b32f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b67801db85d487899eed2e1719a60f8]), partition=
I20240501 19:56:47.036530 27267 tablet_service.cc:1467] Processing CreateTablet for tablet b930ffbf00094a80b6b03b79a1b1b32f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b67801db85d487899eed2e1719a60f8]), partition=
I20240501 19:56:47.036567 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b930ffbf00094a80b6b03b79a1b1b32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.036742 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b930ffbf00094a80b6b03b79a1b1b32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.038228 28289 tablet_bootstrap.cc:492] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:47.038332 28292 tablet_bootstrap.cc:492] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:47.038655 27143 tablet_service.cc:1467] Processing CreateTablet for tablet b930ffbf00094a80b6b03b79a1b1b32f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b67801db85d487899eed2e1719a60f8]), partition=
I20240501 19:56:47.038836 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b930ffbf00094a80b6b03b79a1b1b32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.039203 28289 tablet_bootstrap.cc:654] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.039203 28292 tablet_bootstrap.cc:654] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.040588 28291 tablet_bootstrap.cc:492] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:47.041272 28292 tablet_bootstrap.cc:492] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:47.041347 28292 ts_tablet_manager.cc:1367] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:47.041520 28291 tablet_bootstrap.cc:654] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.041836 28292 raft_consensus.cc:348] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.041945 28292 raft_consensus.cc:374] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.041976 28289 tablet_bootstrap.cc:492] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:47.041977 28292 raft_consensus.cc:729] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.042078 28289 ts_tablet_manager.cc:1367] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:47.042100 28292 consensus_queue.cc:260] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.042376 28292 ts_tablet_manager.cc:1398] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.042557 28289 raft_consensus.cc:348] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.042657 28289 raft_consensus.cc:374] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.042701 28289 raft_consensus.cc:729] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.042819 28289 consensus_queue.cc:260] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.043130 28289 ts_tablet_manager.cc:1398] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.054665 28291 tablet_bootstrap.cc:492] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:47.054795 28291 ts_tablet_manager.cc:1367] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:47.055310 28291 raft_consensus.cc:348] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.055423 28291 raft_consensus.cc:374] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.055459 28291 raft_consensus.cc:729] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.055909 28327 ts_tablet_manager.cc:902] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:47.056201 28291 consensus_queue.cc:260] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.056387 28327 tablet_copy_client.cc:308] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:47.056721 27920 raft_consensus.cc:482] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:47.056797 27920 raft_consensus.cc:504] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.057195 27920 leader_election.cc:288] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:47.058102 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet f89e81297b66402ea09bdb415fb7f385 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:56477)
I20240501 19:56:47.058163 28291 ts_tablet_manager.cc:1398] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:47.058202 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet f89e81297b66402ea09bdb415fb7f385 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:56477: session id = 4a6e54e86a9949e5970b42e8e32e7561-f89e81297b66402ea09bdb415fb7f385
I20240501 19:56:47.058339 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed0e67f44543d09dbeda7100c7e0d6" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:47.058465 27031 raft_consensus.cc:2454] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:47.058734 27359 leader_election.cc:302] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:47.059413 24616 tablet_copy_source_session.cc:205] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:47.060209 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89e81297b66402ea09bdb415fb7f385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.062287 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed0e67f44543d09dbeda7100c7e0d6" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:47.062390 24986 raft_consensus.cc:2454] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:47.062410 28327 tablet_copy_client.cc:783] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:47.062551 28327 tablet_copy_client.cc:647] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:47.060221 27920 raft_consensus.cc:2781] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:47.062892 27920 raft_consensus.cc:482] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:47.062939 27920 raft_consensus.cc:3037] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.063829 28327 tablet_copy_client.cc:523] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:47.063824 27920 raft_consensus.cc:504] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.064536 27920 leader_election.cc:288] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:47.064807 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed0e67f44543d09dbeda7100c7e0d6" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:47.064927 27031 raft_consensus.cc:3037] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.065218 28327 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:47.065877 27031 raft_consensus.cc:2454] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:47.066409 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed0e67f44543d09dbeda7100c7e0d6" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:47.066560 24986 raft_consensus.cc:3037] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.067232 27359 leader_election.cc:302] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:47.068002 24986 raft_consensus.cc:2454] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:47.068308 27920 raft_consensus.cc:2781] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:47.068490 27920 raft_consensus.cc:686] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:47.068663 27920 consensus_queue.cc:237] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.070389 24448 catalog_manager.cc:5484] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:47.088563 27406 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:47.088721 27143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:47.092732 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:47.096166 27011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:47.098022 24574 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:47.108379 24968 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:47.111158 24705 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:56:47.116393 28327 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:47.116796 28327 tablet_bootstrap.cc:492] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:56:47.117223 28327 ts_tablet_manager.cc:1367] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.052s user 0.026s sys 0.000s
I20240501 19:56:47.117756 28327 raft_consensus.cc:348] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:47.117942 28327 raft_consensus.cc:729] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.118131 28327 consensus_queue.cc:260] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:47.120443 28327 ts_tablet_manager.cc:1398] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:47.122371 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-f89e81297b66402ea09bdb415fb7f385 received from {username='slave'} at 127.17.65.8:56477
I20240501 19:56:47.122457 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-f89e81297b66402ea09bdb415fb7f385 on tablet f89e81297b66402ea09bdb415fb7f385 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:47.135085 27267 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:47.181265 27162 raft_consensus.cc:1203] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.181869 27424 raft_consensus.cc:1203] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.215315 27994 ts_tablet_manager.cc:902] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:47.215764 27994 tablet_copy_client.cc:308] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:56:47.216246 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 0f8f81836b9944aa964a8236912c0183 from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:44335)
I20240501 19:56:47.216358 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 0f8f81836b9944aa964a8236912c0183 from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:44335: session id = 53e34bb09d8e41129a5683079ff63610-0f8f81836b9944aa964a8236912c0183
I20240501 19:56:47.217096 24878 tablet_copy_source_session.cc:205] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:47.218678 27994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8f81836b9944aa964a8236912c0183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.221050 27994 tablet_copy_client.cc:783] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:47.221211 27994 tablet_copy_client.cc:647] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:47.223681 27994 tablet_copy_client.cc:523] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:47.224797 27994 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:47.245831 27278 raft_consensus.cc:1203] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.250706 27994 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:47.251106 27994 tablet_bootstrap.cc:492] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:47.251305 27994 ts_tablet_manager.cc:1367] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.003s
I20240501 19:56:47.251830 27994 raft_consensus.cc:348] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:47.251976 27994 raft_consensus.cc:729] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.252112 27994 consensus_queue.cc:260] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:47.252425 27994 ts_tablet_manager.cc:1398] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:47.253121 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-0f8f81836b9944aa964a8236912c0183 received from {username='slave'} at 127.17.65.7:44335
I20240501 19:56:47.253208 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-0f8f81836b9944aa964a8236912c0183 on tablet 0f8f81836b9944aa964a8236912c0183 with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:47.263756 27959 ts_tablet_manager.cc:902] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:47.264261 27959 tablet_copy_client.cc:308] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:47.264753 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet e996e96ce69348c09adc99dc7d9c7125 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:41295)
I20240501 19:56:47.264863 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet e996e96ce69348c09adc99dc7d9c7125 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:41295: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-e996e96ce69348c09adc99dc7d9c7125
I20240501 19:56:47.265836 24616 tablet_copy_source_session.cc:205] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:47.266673 27959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e996e96ce69348c09adc99dc7d9c7125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.268659 27959 tablet_copy_client.cc:783] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:47.268811 27959 tablet_copy_client.cc:647] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:47.269919 27959 tablet_copy_client.cc:523] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:47.270819 27959 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:47.289912 27959 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:47.290251 27959 tablet_bootstrap.cc:492] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:56:47.290445 27959 ts_tablet_manager.cc:1367] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20240501 19:56:47.290813 27959 raft_consensus.cc:348] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.290917 27959 raft_consensus.cc:729] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.291066 27959 consensus_queue.cc:260] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.291313 27959 ts_tablet_manager.cc:1398] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:47.291790 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-e996e96ce69348c09adc99dc7d9c7125 received from {username='slave'} at 127.17.65.5:41295
I20240501 19:56:47.291888 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-e996e96ce69348c09adc99dc7d9c7125 on tablet e996e96ce69348c09adc99dc7d9c7125 with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:47.296337 27639 raft_consensus.cc:482] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:47.296434 27639 raft_consensus.cc:504] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.296797 27639 leader_election.cc:288] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:47.297099 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930ffbf00094a80b6b03b79a1b1b32f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:47.297134 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930ffbf00094a80b6b03b79a1b1b32f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:47.297214 27031 raft_consensus.cc:2454] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:47.297258 27162 raft_consensus.cc:2454] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:47.297482 27227 leader_election.cc:302] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:47.297634 27639 raft_consensus.cc:2781] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:47.297684 27639 raft_consensus.cc:482] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:47.297721 27639 raft_consensus.cc:3037] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.298578 27639 raft_consensus.cc:504] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.298826 27639 leader_election.cc:288] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:47.299146 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930ffbf00094a80b6b03b79a1b1b32f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:47.299154 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930ffbf00094a80b6b03b79a1b1b32f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:47.299234 27031 raft_consensus.cc:3037] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.299258 27162 raft_consensus.cc:3037] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.300006 27031 raft_consensus.cc:2454] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:47.300030 27162 raft_consensus.cc:2454] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:47.300269 27227 leader_election.cc:302] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:47.300444 27639 raft_consensus.cc:2781] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:47.300529 27639 raft_consensus.cc:686] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:47.300668 27639 consensus_queue.cc:237] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.302217 24448 catalog_manager.cc:5484] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:47.320542 24448 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 19:56:47.320703 24448 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 19:56:47.324303 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet bacb09466dc54a5a9286987d1b27d0c0
I20240501 19:56:47.324589 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 1c5b793a014b4d84ba7dd679623fe458
I20240501 19:56:47.324854 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet cfd153394cac410093c6b2f057c3b0de
I20240501 19:56:47.324970 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet bacb09466dc54a5a9286987d1b27d0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:47.325343 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5b793a014b4d84ba7dd679623fe458 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:47.325445 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd153394cac410093c6b2f057c3b0de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:47.325629 27625 tablet_replica.cc:330] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:47.325764 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd153394cac410093c6b2f057c3b0de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:47.325800 27625 raft_consensus.cc:2229] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:47.326028 27625 raft_consensus.cc:2258] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.326246 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5b793a014b4d84ba7dd679623fe458 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:47.326231 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd153394cac410093c6b2f057c3b0de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:47.326409 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet bacb09466dc54a5a9286987d1b27d0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:47.326439 28295 tablet_replica.cc:330] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:47.326416 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5b793a014b4d84ba7dd679623fe458 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:47.326555 27624 tablet_replica.cc:330] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:47.326604 28295 raft_consensus.cc:2229] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.326643 27625 ts_tablet_manager.cc:1875] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.326705 27624 raft_consensus.cc:2229] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.326740 28295 raft_consensus.cc:2258] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.326826 27624 raft_consensus.cc:2258] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.327265 28295 ts_tablet_manager.cc:1875] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.327461 27624 ts_tablet_manager.cc:1875] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.328893 27625 ts_tablet_manager.cc:1888] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.328967 27625 log.cc:1196] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c5b793a014b4d84ba7dd679623fe458
I20240501 19:56:47.329229 27625 ts_tablet_manager.cc:1909] T 1c5b793a014b4d84ba7dd679623fe458 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:47.329444 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet bacb09466dc54a5a9286987d1b27d0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:47.329942 27624 ts_tablet_manager.cc:1888] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.330010 27624 log.cc:1196] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/cfd153394cac410093c6b2f057c3b0de
I20240501 19:56:47.330332 27624 ts_tablet_manager.cc:1909] T cfd153394cac410093c6b2f057c3b0de P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:47.331755 27625 tablet_replica.cc:330] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:47.332365 28295 ts_tablet_manager.cc:1888] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.332314 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet cfd153394cac410093c6b2f057c3b0de (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.332886 27624 tablet_replica.cc:330] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:47.332903 28295 log.cc:1196] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c5b793a014b4d84ba7dd679623fe458
I20240501 19:56:47.333025 27624 raft_consensus.cc:2229] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:47.333199 27624 raft_consensus.cc:2258] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.333391 28295 ts_tablet_manager.cc:1909] T 1c5b793a014b4d84ba7dd679623fe458 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:47.333719 27624 ts_tablet_manager.cc:1875] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.333917 27625 raft_consensus.cc:2229] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:47.334118 27625 raft_consensus.cc:2258] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.334626 27625 ts_tablet_manager.cc:1875] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.335781 28295 tablet_replica.cc:330] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:47.335857 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 1c5b793a014b4d84ba7dd679623fe458 (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.336418 28295 raft_consensus.cc:2229] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.336853 28295 raft_consensus.cc:2258] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.337805 27624 ts_tablet_manager.cc:1888] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.337872 27624 log.cc:1196] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/bacb09466dc54a5a9286987d1b27d0c0
I20240501 19:56:47.338117 27624 ts_tablet_manager.cc:1909] T bacb09466dc54a5a9286987d1b27d0c0 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:47.338117 28295 ts_tablet_manager.cc:1875] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.338876 27624 tablet_replica.cc:330] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:47.338949 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet bacb09466dc54a5a9286987d1b27d0c0 (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.339193 27624 raft_consensus.cc:2229] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.339226 27011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:47.339365 27624 raft_consensus.cc:2258] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.339809 27624 ts_tablet_manager.cc:1875] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.340217 27625 ts_tablet_manager.cc:1888] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.340279 27625 log.cc:1196] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/cfd153394cac410093c6b2f057c3b0de
I20240501 19:56:47.340580 27625 ts_tablet_manager.cc:1909] T cfd153394cac410093c6b2f057c3b0de P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:47.341838 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 1c5b793a014b4d84ba7dd679623fe458 (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.341985 27624 ts_tablet_manager.cc:1888] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.341989 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet cfd153394cac410093c6b2f057c3b0de (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.342048 27624 log.cc:1196] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c5b793a014b4d84ba7dd679623fe458
I20240501 19:56:47.342300 27624 ts_tablet_manager.cc:1909] T 1c5b793a014b4d84ba7dd679623fe458 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:47.342414 27406 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:47.341799 28295 ts_tablet_manager.cc:1888] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.342722 28295 log.cc:1196] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/bacb09466dc54a5a9286987d1b27d0c0
I20240501 19:56:47.343163 28295 ts_tablet_manager.cc:1909] T bacb09466dc54a5a9286987d1b27d0c0 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:47.343623 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 1c5b793a014b4d84ba7dd679623fe458 (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.344029 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet bacb09466dc54a5a9286987d1b27d0c0 (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.348124 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:56:47.348434 28466 tablet_replica.cc:330] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:47.348721 28466 raft_consensus.cc:2229] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.348933 28466 raft_consensus.cc:2258] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.349077 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:47.349583 28466 ts_tablet_manager.cc:1875] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.351893 28466 ts_tablet_manager.cc:1888] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.352234 28466 log.cc:1196] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/bacb09466dc54a5a9286987d1b27d0c0
I20240501 19:56:47.353078 24574 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:47.353130 28466 ts_tablet_manager.cc:1909] T bacb09466dc54a5a9286987d1b27d0c0 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:47.354089 28466 tablet_replica.cc:330] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:47.354193 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet bacb09466dc54a5a9286987d1b27d0c0 (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.354254 28466 raft_consensus.cc:2229] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.354452 28466 raft_consensus.cc:2258] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.359992 27267 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:47.362272 27143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:47.372941 28466 ts_tablet_manager.cc:1875] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.375121 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:47.375221 27639 consensus_queue.cc:1035] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.376842 28466 ts_tablet_manager.cc:1888] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.376919 28466 log.cc:1196] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/cfd153394cac410093c6b2f057c3b0de
I20240501 19:56:47.377197 28466 ts_tablet_manager.cc:1909] T cfd153394cac410093c6b2f057c3b0de P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:47.378365 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet cfd153394cac410093c6b2f057c3b0de (table rebalance_test_table_8 [id=93e453e991024af88050d72a55aba4c9]) successfully deleted
I20240501 19:56:47.383690 27032 raft_consensus.cc:1203] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.399433 27278 raft_consensus.cc:1203] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.410014 27162 raft_consensus.cc:3037] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.411711 28334 consensus_queue.cc:1035] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.426623 27424 raft_consensus.cc:1203] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.438151 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:47.439234 24727 consensus_queue.cc:237] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:47.440591 27278 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:47.441403 27637 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:47.442417 24595 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:47.443106 27031 raft_consensus.cc:1261] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:47.443177 27162 raft_consensus.cc:1261] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:47.443414 26677 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:47.443508 28335 consensus_queue.cc:1035] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.443632 28334 consensus_queue.cc:1035] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.443841 26677 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:47.444351 27278 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:47.446602 24448 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet b930ffbf00094a80b6b03b79a1b1b32f (table rebalancer_dynamic_table [id=5b67801db85d487899eed2e1719a60f8]). Expected version 1 got 0
I20240501 19:56:47.447050 24446 catalog_manager.cc:5484] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:47.447789 24448 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet b930ffbf00094a80b6b03b79a1b1b32f (table rebalancer_dynamic_table [id=5b67801db85d487899eed2e1719a60f8]). Expected version 1 got 0
I20240501 19:56:47.448709 28493 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:47.448734 24595 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:47.449023 28493 tablet.cc:1620] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:47.449704 28492 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:47.450469 28492 tablet.cc:1620] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:47.453183 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=5b67801db85d487899eed2e1719a60f8] alter complete (version 1)
I20240501 19:56:47.454716 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b240b461b548a087fcabdeb938df1e with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:47.454957 28466 tablet_replica.cc:330] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:47.455142 28466 raft_consensus.cc:2229] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.455335 28466 raft_consensus.cc:2258] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.455875 28466 ts_tablet_manager.cc:1875] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:47.456923 27162 raft_consensus.cc:1203] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.458009 28466 ts_tablet_manager.cc:1888] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:47.458070 28466 log.cc:1196] T a4b240b461b548a087fcabdeb938df1e P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/a4b240b461b548a087fcabdeb938df1e
I20240501 19:56:47.458058 28499 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:47.458187 27424 raft_consensus.cc:1203] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.458295 28499 tablet.cc:1620] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:47.458918 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet a4b240b461b548a087fcabdeb938df1e (table rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]) successfully deleted
I20240501 19:56:47.459769 24448 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet b930ffbf00094a80b6b03b79a1b1b32f (table rebalancer_dynamic_table [id=5b67801db85d487899eed2e1719a60f8]). Expected version 1 got 0
I20240501 19:56:47.480482 27031 raft_consensus.cc:1203] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.490278 28075 consensus_queue.cc:1035] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.495348 28518 consensus_queue.cc:1035] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.521699 27920 consensus_queue.cc:1035] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.525687 28389 consensus_queue.cc:1035] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.541903 27143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:47.545253 27011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:47.547857 24574 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:47.549000 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:47.549955 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:56:47.552326 27406 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:47.558354 27267 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:47.562656 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:47.573226 28492 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:47.573563 28492 tablet.cc:1620] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:47.575093 28493 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:47.575368 28493 tablet.cc:1620] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:47.583225 28499 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:47.584198 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:47.585521 24452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:47.585907 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=5b67801db85d487899eed2e1719a60f8] alter complete (version 2)
I20240501 19:56:47.586081 28499 tablet.cc:1620] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:47.593318 27143 tablet_service.cc:1467] Processing CreateTablet for tablet afc479e18b5e4efa8a8fa113db3dbf1b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.593598 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc479e18b5e4efa8a8fa113db3dbf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.593806 27011 tablet_service.cc:1467] Processing CreateTablet for tablet fce9f9c2a5be47d3986e3d7b53570b12 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.594049 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce9f9c2a5be47d3986e3d7b53570b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.596283 27405 tablet_service.cc:1467] Processing CreateTablet for tablet fce9f9c2a5be47d3986e3d7b53570b12 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.596309 27274 tablet_service.cc:1467] Processing CreateTablet for tablet fce9f9c2a5be47d3986e3d7b53570b12 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.596577 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce9f9c2a5be47d3986e3d7b53570b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.596585 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce9f9c2a5be47d3986e3d7b53570b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.596751 24966 tablet_service.cc:1467] Processing CreateTablet for tablet c6a7cb30f31b48b49115f8f016c09d82 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.596774 24968 tablet_service.cc:1467] Processing CreateTablet for tablet afc479e18b5e4efa8a8fa113db3dbf1b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.597028 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a7cb30f31b48b49115f8f016c09d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.597134 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc479e18b5e4efa8a8fa113db3dbf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.597406 27406 tablet_service.cc:1467] Processing CreateTablet for tablet afc479e18b5e4efa8a8fa113db3dbf1b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.597604 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc479e18b5e4efa8a8fa113db3dbf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.599805 27267 tablet_service.cc:1467] Processing CreateTablet for tablet c6a7cb30f31b48b49115f8f016c09d82 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.600003 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a7cb30f31b48b49115f8f016c09d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.602458 24707 tablet_service.cc:1467] Processing CreateTablet for tablet c6a7cb30f31b48b49115f8f016c09d82 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:47.602721 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a7cb30f31b48b49115f8f016c09d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.604903 28555 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:47.605830 28555 tablet_bootstrap.cc:654] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.606053 28556 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:47.616202 28556 tablet_bootstrap.cc:654] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.617008 28559 tablet_bootstrap.cc:492] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:47.617978 28559 tablet_bootstrap.cc:654] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.621537 28556 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:47.621696 28556 ts_tablet_manager.cc:1367] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:56:47.623225 28556 raft_consensus.cc:348] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.623450 28556 raft_consensus.cc:374] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.623690 28556 raft_consensus.cc:729] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.624089 28556 consensus_queue.cc:260] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.625198 28556 ts_tablet_manager.cc:1398] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:47.625402 28556 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:47.628831 28556 tablet_bootstrap.cc:654] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.629984 28556 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:47.630113 28556 ts_tablet_manager.cc:1367] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:47.630872 28556 raft_consensus.cc:348] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.631039 28556 raft_consensus.cc:374] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.631316 28556 raft_consensus.cc:729] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.631479 28556 consensus_queue.cc:260] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.633121 28557 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:47.633881 28557 tablet_bootstrap.cc:654] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.634436 28556 ts_tablet_manager.cc:1398] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:47.635221 28558 tablet_bootstrap.cc:492] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:47.636241 28554 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:47.637094 28554 tablet_bootstrap.cc:654] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.637634 28558 tablet_bootstrap.cc:654] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.638667 28558 tablet_bootstrap.cc:492] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:47.638754 28558 ts_tablet_manager.cc:1367] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:47.639211 28558 raft_consensus.cc:348] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.639309 28558 raft_consensus.cc:374] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.639336 28558 raft_consensus.cc:729] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.639519 28558 consensus_queue.cc:260] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.640492 28558 ts_tablet_manager.cc:1398] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:47.640763 28558 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:47.641616 28558 tablet_bootstrap.cc:654] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.643011 28557 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:47.643184 28557 ts_tablet_manager.cc:1367] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:56:47.643754 28557 raft_consensus.cc:348] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.643931 28557 raft_consensus.cc:374] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.644011 28557 raft_consensus.cc:729] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.644209 28557 consensus_queue.cc:260] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.644595 28557 ts_tablet_manager.cc:1398] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:47.645020 28558 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:47.645095 28558 ts_tablet_manager.cc:1367] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:47.645527 28558 raft_consensus.cc:348] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.645649 28558 raft_consensus.cc:374] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.645684 28558 raft_consensus.cc:729] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.645789 28558 consensus_queue.cc:260] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.646041 28558 ts_tablet_manager.cc:1398] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.647207 28557 tablet_bootstrap.cc:492] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:47.648712 28557 tablet_bootstrap.cc:654] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.650168 28557 tablet_bootstrap.cc:492] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:47.650305 28557 ts_tablet_manager.cc:1367] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:47.650785 28557 raft_consensus.cc:348] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.650902 28557 raft_consensus.cc:374] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.650960 28557 raft_consensus.cc:729] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.651124 28557 consensus_queue.cc:260] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.651443 28557 ts_tablet_manager.cc:1398] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.652622 27278 raft_consensus.cc:1203] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.654747 24725 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f"
dest_uuid: "9081292597be4a72a42b216fddcd7971"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54280
I20240501 19:56:47.654918 24725 raft_consensus.cc:595] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:47.657949 28555 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:47.658057 28555 ts_tablet_manager.cc:1367] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20240501 19:56:47.658478 28555 raft_consensus.cc:348] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.658587 28555 raft_consensus.cc:374] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.658627 28555 raft_consensus.cc:729] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.658748 28555 consensus_queue.cc:260] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.659446 28554 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:47.659535 28555 ts_tablet_manager.cc:1398] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.659555 28554 ts_tablet_manager.cc:1367] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 19:56:47.659996 28554 raft_consensus.cc:348] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.660125 28554 raft_consensus.cc:374] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.660157 28554 raft_consensus.cc:729] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.660280 28554 consensus_queue.cc:260] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.660429 28559 tablet_bootstrap.cc:492] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:47.660511 28559 ts_tablet_manager.cc:1367] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240501 19:56:47.660576 28554 ts_tablet_manager.cc:1398] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.660918 28559 raft_consensus.cc:348] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.661031 28559 raft_consensus.cc:374] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.661069 28559 raft_consensus.cc:729] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.661219 28559 consensus_queue.cc:260] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.661522 28559 ts_tablet_manager.cc:1398] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.672010 28518 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:47.672116 28518 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.672585 28518 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:47.672950 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc479e18b5e4efa8a8fa113db3dbf1b" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:47.673100 27424 raft_consensus.cc:2454] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:47.673583 27098 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:47.673802 28518 raft_consensus.cc:2781] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:47.673859 28518 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:47.673907 28518 raft_consensus.cc:3037] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.674973 28518 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.675379 28518 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:47.675662 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc479e18b5e4efa8a8fa113db3dbf1b" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:47.675776 27424 raft_consensus.cc:3037] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.676743 27424 raft_consensus.cc:2454] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:47.677096 27098 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:47.677222 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc479e18b5e4efa8a8fa113db3dbf1b" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:47.677318 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc479e18b5e4efa8a8fa113db3dbf1b" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:47.677348 24986 raft_consensus.cc:2454] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:47.678838 28518 raft_consensus.cc:2781] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:47.678930 28518 raft_consensus.cc:686] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:47.679255 28518 consensus_queue.cc:237] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.681075 24452 catalog_manager.cc:5484] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:47.681288 28335 raft_consensus.cc:482] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:47.681398 28335 raft_consensus.cc:504] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.681825 28335 leader_election.cc:288] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:47.682157 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a7cb30f31b48b49115f8f016c09d82" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:47.682232 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a7cb30f31b48b49115f8f016c09d82" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:47.682293 24725 raft_consensus.cc:2454] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:47.682365 24986 raft_consensus.cc:2454] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:47.682629 27227 leader_election.cc:302] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:47.683001 28335 raft_consensus.cc:2781] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:47.683080 28335 raft_consensus.cc:482] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:47.683117 28335 raft_consensus.cc:3037] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.684104 28335 raft_consensus.cc:504] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.684448 28335 leader_election.cc:288] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:47.684798 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a7cb30f31b48b49115f8f016c09d82" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:47.684811 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a7cb30f31b48b49115f8f016c09d82" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:47.684902 24986 raft_consensus.cc:3037] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.684918 24725 raft_consensus.cc:3037] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.685647 24986 raft_consensus.cc:2454] T c6a7cb30f31b48b49115f8f016c09d82 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:47.685843 24725 raft_consensus.cc:2454] T c6a7cb30f31b48b49115f8f016c09d82 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:47.685900 27228 leader_election.cc:302] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:47.686057 28335 raft_consensus.cc:2781] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:47.686143 28335 raft_consensus.cc:686] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:47.686281 28335 consensus_queue.cc:237] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:47.688068 24452 catalog_manager.cc:5484] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:47.691329 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:47.696282 28492 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:47.696930 28493 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:47.699712 28499 tablet.cc:1722] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:47.703229 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=5b67801db85d487899eed2e1719a60f8] alter complete (version 3)
I20240501 19:56:47.799476 27031 raft_consensus.cc:1203] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:47.806901 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:47.807067 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:47.809123 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet b930ffbf00094a80b6b03b79a1b1b32f
I20240501 19:56:47.809664 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet b930ffbf00094a80b6b03b79a1b1b32f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:56:47.809942 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet b930ffbf00094a80b6b03b79a1b1b32f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:47.809952 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet b930ffbf00094a80b6b03b79a1b1b32f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:47.809993 28614 tablet_replica.cc:330] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:56:47.810134 28614 raft_consensus.cc:2229] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:47.810220 28615 tablet_replica.cc:330] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:47.810233 28616 tablet_replica.cc:330] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:47.810410 28615 raft_consensus.cc:2229] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.810442 28616 raft_consensus.cc:2229] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.810529 28614 raft_consensus.cc:2258] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.810588 28615 raft_consensus.cc:2258] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.810603 28616 raft_consensus.cc:2258] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.811188 28614 ts_tablet_manager.cc:1875] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.811199 28615 ts_tablet_manager.cc:1875] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.811199 28616 ts_tablet_manager.cc:1875] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.812713 27920 raft_consensus.cc:482] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:47.812824 27920 raft_consensus.cc:504] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.813079 28614 ts_tablet_manager.cc:1888] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:47.813140 28614 log.cc:1196] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/b930ffbf00094a80b6b03b79a1b1b32f
I20240501 19:56:47.813354 28615 ts_tablet_manager.cc:1888] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:47.813354 28616 ts_tablet_manager.cc:1888] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:47.813417 28614 ts_tablet_manager.cc:1909] T b930ffbf00094a80b6b03b79a1b1b32f P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:56:47.813421 28615 log.cc:1196] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/b930ffbf00094a80b6b03b79a1b1b32f
I20240501 19:56:47.813405 27920 leader_election.cc:288] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:47.813427 28616 log.cc:1196] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/b930ffbf00094a80b6b03b79a1b1b32f
I20240501 19:56:47.813691 28616 ts_tablet_manager.cc:1909] T b930ffbf00094a80b6b03b79a1b1b32f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:47.813695 28615 ts_tablet_manager.cc:1909] T b930ffbf00094a80b6b03b79a1b1b32f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:47.814143 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet b930ffbf00094a80b6b03b79a1b1b32f (table rebalancer_dynamic_table_new_name [id=5b67801db85d487899eed2e1719a60f8]) successfully deleted
I20240501 19:56:47.814723 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce9f9c2a5be47d3986e3d7b53570b12" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:47.814844 27031 raft_consensus.cc:2454] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:47.815177 27359 leader_election.cc:302] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:47.815275 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet b930ffbf00094a80b6b03b79a1b1b32f (table rebalancer_dynamic_table_new_name [id=5b67801db85d487899eed2e1719a60f8]) successfully deleted
I20240501 19:56:47.815374 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet b930ffbf00094a80b6b03b79a1b1b32f (table rebalancer_dynamic_table_new_name [id=5b67801db85d487899eed2e1719a60f8]) successfully deleted
I20240501 19:56:47.815528 27920 raft_consensus.cc:2781] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:47.815587 27920 raft_consensus.cc:482] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:47.815618 27920 raft_consensus.cc:3037] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.816326 27920 raft_consensus.cc:504] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.816645 27920 leader_election.cc:288] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:47.816941 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce9f9c2a5be47d3986e3d7b53570b12" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:47.817065 27031 raft_consensus.cc:3037] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:47.817302 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce9f9c2a5be47d3986e3d7b53570b12" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:47.817366 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce9f9c2a5be47d3986e3d7b53570b12" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:47.817420 27278 raft_consensus.cc:2454] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:47.818002 27031 raft_consensus.cc:2454] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:47.818259 27359 leader_election.cc:302] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters: 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:47.818428 27920 raft_consensus.cc:2781] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:47.818529 27920 raft_consensus.cc:686] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:47.818646 27920 consensus_queue.cc:237] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:47.820428 24454 catalog_manager.cc:5484] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:47.822055 27637 raft_consensus.cc:979] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: : Instructing follower 4a6e54e86a9949e5970b42e8e32e7561 to start an election
I20240501 19:56:47.822150 26677 raft_consensus.cc:1067] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Signalling peer 4a6e54e86a9949e5970b42e8e32e7561 to start an election
I20240501 19:56:47.822788 27425 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f"
dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
from {username='slave'} at 127.17.65.2:59691
I20240501 19:56:47.822934 27425 raft_consensus.cc:482] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:47.822995 27425 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:47.823732 27425 raft_consensus.cc:504] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:47.824136 27425 leader_election.cc:288] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 2 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:47.824465 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:47.824592 24856 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:47.824568 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:47.824568 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:47.824684 24986 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:47.824678 24725 raft_consensus.cc:3032] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:47.824720 24725 raft_consensus.cc:729] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:47.824884 24725 consensus_queue.cc:260] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:47.825047 24725 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:47.825388 24986 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 2.
I20240501 19:56:47.825587 24856 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 2.
I20240501 19:56:47.825914 27360 leader_election.cc:302] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:47.826050 24725 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 2.
I20240501 19:56:47.826076 28389 raft_consensus.cc:2781] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:47.826138 28389 raft_consensus.cc:686] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:47.826285 28389 consensus_queue.cc:237] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:47.828109 24454 catalog_manager.cc:5484] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 1 to 2, leader changed from 9081292597be4a72a42b216fddcd7971 (127.17.65.2) to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 2 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:47.830226 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 19:56:47.830332 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 19:56:47.832248 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet e2353c8b5f284fd3bd1c5f25e2eabe8f
I20240501 19:56:47.832515 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet dd88b75ba5ff41cdbe02f2b83d949681
I20240501 19:56:47.832738 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 2b02546b0c8c404c919ae9998984188f
I20240501 19:56:47.832829 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:47.832837 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:47.832842 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet e2353c8b5f284fd3bd1c5f25e2eabe8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:47.833006 28295 tablet_replica.cc:330] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:47.833006 27625 tablet_replica.cc:330] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:47.833129 27625 raft_consensus.cc:2229] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.833151 28295 raft_consensus.cc:2229] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.833273 27625 raft_consensus.cc:2258] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.833273 28295 raft_consensus.cc:2258] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.833787 27625 ts_tablet_manager.cc:1875] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.833787 28295 ts_tablet_manager.cc:1875] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.836274 27625 ts_tablet_manager.cc:1888] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.836344 27625 log.cc:1196] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/e2353c8b5f284fd3bd1c5f25e2eabe8f
I20240501 19:56:47.836628 27625 ts_tablet_manager.cc:1909] T e2353c8b5f284fd3bd1c5f25e2eabe8f P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:47.837340 27624 tablet_replica.cc:330] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:47.837479 28295 ts_tablet_manager.cc:1888] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.837507 27624 raft_consensus.cc:2229] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:47.837540 28295 log.cc:1196] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/e2353c8b5f284fd3bd1c5f25e2eabe8f
I20240501 19:56:47.837738 27624 raft_consensus.cc:2258] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.837810 28295 ts_tablet_manager.cc:1909] T e2353c8b5f284fd3bd1c5f25e2eabe8f P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:47.838265 27624 ts_tablet_manager.cc:1875] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.838768 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet e2353c8b5f284fd3bd1c5f25e2eabe8f (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.838956 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b02546b0c8c404c919ae9998984188f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:47.839164 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet dd88b75ba5ff41cdbe02f2b83d949681 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:47.839211 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b02546b0c8c404c919ae9998984188f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:47.839216 28295 tablet_replica.cc:330] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:47.839357 28295 raft_consensus.cc:2229] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:47.839550 28295 raft_consensus.cc:2258] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.840066 28295 ts_tablet_manager.cc:1875] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.840456 27624 ts_tablet_manager.cc:1888] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.840514 27624 log.cc:1196] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/e2353c8b5f284fd3bd1c5f25e2eabe8f
I20240501 19:56:47.840763 27624 ts_tablet_manager.cc:1909] T e2353c8b5f284fd3bd1c5f25e2eabe8f P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:47.841344 27624 tablet_replica.cc:330] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:47.841470 27624 raft_consensus.cc:2229] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:47.841451 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet e2353c8b5f284fd3bd1c5f25e2eabe8f (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.841650 27624 raft_consensus.cc:2258] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.842000 28295 ts_tablet_manager.cc:1888] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.842057 28295 log.cc:1196] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/2b02546b0c8c404c919ae9998984188f
I20240501 19:56:47.842147 27624 ts_tablet_manager.cc:1875] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.842308 28295 ts_tablet_manager.cc:1909] T 2b02546b0c8c404c919ae9998984188f P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:47.843250 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet e2353c8b5f284fd3bd1c5f25e2eabe8f (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.843693 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet dd88b75ba5ff41cdbe02f2b83d949681 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:47.843887 28466 tablet_replica.cc:330] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:47.844028 28466 raft_consensus.cc:2229] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.844184 28466 raft_consensus.cc:2258] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.844465 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b02546b0c8c404c919ae9998984188f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:47.844558 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet dd88b75ba5ff41cdbe02f2b83d949681 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:47 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:47.844729 28466 ts_tablet_manager.cc:1875] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.845028 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 2b02546b0c8c404c919ae9998984188f (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.845180 27625 tablet_replica.cc:330] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:47.845335 27625 raft_consensus.cc:2229] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.845490 27625 raft_consensus.cc:2258] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.846959 28466 ts_tablet_manager.cc:1888] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.847172 27625 ts_tablet_manager.cc:1875] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.848050 28466 log.cc:1196] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/dd88b75ba5ff41cdbe02f2b83d949681
I20240501 19:56:47.848326 27624 ts_tablet_manager.cc:1888] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.848394 28466 ts_tablet_manager.cc:1909] T dd88b75ba5ff41cdbe02f2b83d949681 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:47.848407 27624 log.cc:1196] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd88b75ba5ff41cdbe02f2b83d949681
I20240501 19:56:47.848745 27624 ts_tablet_manager.cc:1909] T dd88b75ba5ff41cdbe02f2b83d949681 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:47.850003 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet dd88b75ba5ff41cdbe02f2b83d949681 (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.850546 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet dd88b75ba5ff41cdbe02f2b83d949681 (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.850872 27624 tablet_replica.cc:330] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:47.853404 27625 ts_tablet_manager.cc:1888] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.853530 27625 log.cc:1196] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/2b02546b0c8c404c919ae9998984188f
I20240501 19:56:47.853883 27625 ts_tablet_manager.cc:1909] T 2b02546b0c8c404c919ae9998984188f P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:47.854599 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 2b02546b0c8c404c919ae9998984188f (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.854684 27011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:47.855015 27625 tablet_replica.cc:330] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:47.855193 27625 raft_consensus.cc:2229] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.855391 27625 raft_consensus.cc:2258] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.856576 27625 ts_tablet_manager.cc:1875] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.859711 27624 raft_consensus.cc:2229] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:47.859896 27624 raft_consensus.cc:2258] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:47.860538 27624 ts_tablet_manager.cc:1875] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:47.861101 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:47.865916 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:47.866266 27625 ts_tablet_manager.cc:1888] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.866421 27625 log.cc:1196] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/dd88b75ba5ff41cdbe02f2b83d949681
I20240501 19:56:47.868376 27625 ts_tablet_manager.cc:1909] T dd88b75ba5ff41cdbe02f2b83d949681 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:47.868602 27624 ts_tablet_manager.cc:1888] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:47.868685 27624 log.cc:1196] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b02546b0c8c404c919ae9998984188f
I20240501 19:56:47.868948 27624 ts_tablet_manager.cc:1909] T 2b02546b0c8c404c919ae9998984188f P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:47.869326 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet dd88b75ba5ff41cdbe02f2b83d949681 (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.873330 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 2b02546b0c8c404c919ae9998984188f (table rebalance_test_table_10 [id=5c1bf034787b4d7ab64ef1c0ecc34d24]) successfully deleted
I20240501 19:56:47.874136 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:47.874393 24574 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:47.874467 27274 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:47.877408 27143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:47.880564 24705 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:47.911347 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:47.916905 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 03f6fc63ecb1457486fbabacab60d2fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0c7f223593a4187bdb5f36c9339d403]), partition=
I20240501 19:56:47.916905 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 03f6fc63ecb1457486fbabacab60d2fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0c7f223593a4187bdb5f36c9339d403]), partition=
I20240501 19:56:47.917166 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f6fc63ecb1457486fbabacab60d2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.917171 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f6fc63ecb1457486fbabacab60d2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.919379 28559 tablet_bootstrap.cc:492] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:47.920256 28559 tablet_bootstrap.cc:654] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.921113 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 03f6fc63ecb1457486fbabacab60d2fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0c7f223593a4187bdb5f36c9339d403]), partition=
I20240501 19:56:47.921325 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f6fc63ecb1457486fbabacab60d2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.923388 28558 tablet_bootstrap.cc:492] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:47.924111 28558 tablet_bootstrap.cc:654] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.924892 28559 tablet_bootstrap.cc:492] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:47.924968 28559 ts_tablet_manager.cc:1367] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:47.925359 28559 raft_consensus.cc:348] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:47.925459 28559 raft_consensus.cc:374] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.925484 28559 raft_consensus.cc:729] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.925598 28559 consensus_queue.cc:260] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:47.925840 28559 ts_tablet_manager.cc:1398] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.926846 28558 tablet_bootstrap.cc:492] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:47.926914 28558 ts_tablet_manager.cc:1367] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:47.926944 28631 tablet_bootstrap.cc:492] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:47.927317 28558 raft_consensus.cc:348] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:47.927423 28558 raft_consensus.cc:374] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.927474 28558 raft_consensus.cc:729] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.927572 28558 consensus_queue.cc:260] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:47.927827 28558 ts_tablet_manager.cc:1398] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:47.928256 28631 tablet_bootstrap.cc:654] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:47.929965 28631 tablet_bootstrap.cc:492] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:47.930100 28631 ts_tablet_manager.cc:1367] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:47.930665 28631 raft_consensus.cc:348] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:47.930775 28631 raft_consensus.cc:374] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:47.930857 28631 raft_consensus.cc:729] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:47.930990 28631 consensus_queue.cc:260] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:47.931352 28631 ts_tablet_manager.cc:1398] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:48.065979 24986 raft_consensus.cc:3037] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.067243 28518 consensus_queue.cc:1035] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.070081 28518 consensus_queue.cc:1035] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.102756 26677 raft_consensus.cc:482] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:48.102870 26677 raft_consensus.cc:504] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:48.103323 26677 leader_election.cc:288] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:48.103663 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f6fc63ecb1457486fbabacab60d2fb" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:48.103669 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f6fc63ecb1457486fbabacab60d2fb" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:48.103807 24595 raft_consensus.cc:2454] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:48.103814 27276 raft_consensus.cc:2454] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:48.104097 24662 leader_election.cc:302] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:48.104290 27637 raft_consensus.cc:2781] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:48.104349 27637 raft_consensus.cc:482] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:48.104413 27637 raft_consensus.cc:3037] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.105294 27637 raft_consensus.cc:504] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:48.105623 27637 leader_election.cc:288] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:48.105916 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f6fc63ecb1457486fbabacab60d2fb" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:48.105926 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f6fc63ecb1457486fbabacab60d2fb" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:48.106026 24595 raft_consensus.cc:3037] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.106038 27276 raft_consensus.cc:3037] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.107017 27276 raft_consensus.cc:2454] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:48.107082 24595 raft_consensus.cc:2454] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:48.107344 24660 leader_election.cc:302] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:48.107486 27637 raft_consensus.cc:2781] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:48.107575 27637 raft_consensus.cc:686] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:48.107724 27637 consensus_queue.cc:237] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:48.109910 24448 catalog_manager.cc:5484] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:48.133780 27143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:48.135449 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:48.139573 27011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:48.152372 27274 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:48.153596 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:48.155916 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:48.175631 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:48.177464 27639 consensus_queue.cc:1035] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.185050 28334 consensus_queue.cc:1035] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.188035 24574 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:48.206127 24856 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:48.207511 28389 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:48.217958 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:48.223177 24595 raft_consensus.cc:1261] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:48.223178 27276 raft_consensus.cc:1261] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:48.223834 27637 consensus_queue.cc:1035] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.225265 27637 consensus_queue.cc:1035] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.227198 24448 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 03f6fc63ecb1457486fbabacab60d2fb (table rebalancer_dynamic_table [id=a0c7f223593a4187bdb5f36c9339d403]). Expected version 1 got 0
I20240501 19:56:48.228992 24448 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet 03f6fc63ecb1457486fbabacab60d2fb (table rebalancer_dynamic_table [id=a0c7f223593a4187bdb5f36c9339d403]). Expected version 1 got 0
I20240501 19:56:48.229805 28715 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:48.230814 28715 tablet.cc:1620] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:48.231233 28710 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:48.230501 28717 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:48.231670 28710 tablet.cc:1620] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:48.231751 28717 tablet.cc:1620] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:48.232224 24448 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 03f6fc63ecb1457486fbabacab60d2fb (table rebalancer_dynamic_table [id=a0c7f223593a4187bdb5f36c9339d403]). Expected version 1 got 0
I20240501 19:56:48.235661 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=a0c7f223593a4187bdb5f36c9339d403] alter complete (version 1)
I20240501 19:56:48.239192 24856 consensus_queue.cc:237] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.240509 24595 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:48.240542 27162 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:48.240904 27484 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:48.241088 28114 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:48.242693 27484 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:48.243510 24595 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:48.243544 27162 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:48.245491 27276 raft_consensus.cc:3037] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.246058 24446 catalog_manager.cc:5484] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:48.247016 28389 consensus_queue.cc:1035] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.249887 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2603a318c34256a9fa28979ae41fec with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:48.250193 27625 tablet_replica.cc:330] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:48.250339 27625 raft_consensus.cc:2229] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.250473 27625 raft_consensus.cc:2258] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.251019 27625 ts_tablet_manager.cc:1875] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:48.251168 27920 consensus_queue.cc:1035] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.251853 24727 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:48.252344 28389 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:48.253248 27625 ts_tablet_manager.cc:1888] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:48.253316 27625 log.cc:1196] T 6d2603a318c34256a9fa28979ae41fec P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/6d2603a318c34256a9fa28979ae41fec
I20240501 19:56:48.253814 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 6d2603a318c34256a9fa28979ae41fec (table rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]) successfully deleted
I20240501 19:56:48.254812 24987 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:48.256739 28729 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:48.333917 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:48.336575 27143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:48.336701 24445 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 19:56:48.336859 24445 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 19:56:48.338158 27274 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:48.339697 24445 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 00b49c7daa8e49e9b264fdcab76ea976
I20240501 19:56:48.339995 24445 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 4 replicas of tablet f6dbe23d20b34a348938dcc3571020b7
I20240501 19:56:48.340224 24445 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 4 replicas of tablet 09d33aa5ebfd404ca218b069767d216b
I20240501 19:56:48.340791 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b49c7daa8e49e9b264fdcab76ea976 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:48.341154 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet f6dbe23d20b34a348938dcc3571020b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:48.341307 27625 tablet_replica.cc:330] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:48.341331 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d33aa5ebfd404ca218b069767d216b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:48.341426 27625 raft_consensus.cc:2229] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.341557 27625 raft_consensus.cc:2258] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.342053 27625 ts_tablet_manager.cc:1875] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.343704 28295 tablet_replica.cc:330] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:48.343842 28295 raft_consensus.cc:2229] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.343962 28295 raft_consensus.cc:2258] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.344439 27625 ts_tablet_manager.cc:1888] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.344504 27625 log.cc:1196] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/f6dbe23d20b34a348938dcc3571020b7
I20240501 19:56:48.344528 28295 ts_tablet_manager.cc:1875] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.344785 27625 ts_tablet_manager.cc:1909] T f6dbe23d20b34a348938dcc3571020b7 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:48.345434 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d33aa5ebfd404ca218b069767d216b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:48.345731 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b49c7daa8e49e9b264fdcab76ea976 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:48.345762 28779 tablet_replica.cc:330] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:48.345894 28779 raft_consensus.cc:2229] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.345904 27624 tablet_replica.cc:330] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:48.345984 28779 raft_consensus.cc:2258] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.346035 27624 raft_consensus.cc:2229] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:48.346252 27624 raft_consensus.cc:2258] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.346581 28779 ts_tablet_manager.cc:1875] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.346643 28295 ts_tablet_manager.cc:1888] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:48.346705 28295 log.cc:1196] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/00b49c7daa8e49e9b264fdcab76ea976
I20240501 19:56:48.346817 27624 ts_tablet_manager.cc:1875] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.346967 28295 ts_tablet_manager.cc:1909] T 00b49c7daa8e49e9b264fdcab76ea976 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:48.347138 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet f6dbe23d20b34a348938dcc3571020b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:48.347262 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d33aa5ebfd404ca218b069767d216b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:48.347457 24968 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:48.347599 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet f6dbe23d20b34a348938dcc3571020b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:48.347662 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 00b49c7daa8e49e9b264fdcab76ea976 (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.348987 27624 ts_tablet_manager.cc:1888] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:48.349045 27624 log.cc:1196] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/00b49c7daa8e49e9b264fdcab76ea976
I20240501 19:56:48.349282 27624 ts_tablet_manager.cc:1909] T 00b49c7daa8e49e9b264fdcab76ea976 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:48.349941 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b49c7daa8e49e9b264fdcab76ea976 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:48.350008 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d33aa5ebfd404ca218b069767d216b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:48.350144 28466 tablet_replica.cc:330] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:48.350281 28466 raft_consensus.cc:2229] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.350276 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet f6dbe23d20b34a348938dcc3571020b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:48.350431 28466 raft_consensus.cc:2258] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.350442 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet f6dbe23d20b34a348938dcc3571020b7 (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.350920 28466 ts_tablet_manager.cc:1875] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.353065 28466 ts_tablet_manager.cc:1888] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:48.353140 28466 log.cc:1196] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/00b49c7daa8e49e9b264fdcab76ea976
I20240501 19:56:48.353428 28466 ts_tablet_manager.cc:1909] T 00b49c7daa8e49e9b264fdcab76ea976 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:48.353988 28466 tablet_replica.cc:330] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:48.354076 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 00b49c7daa8e49e9b264fdcab76ea976 (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.354107 28466 raft_consensus.cc:2229] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.354386 27625 tablet_replica.cc:330] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:48.354245 28466 raft_consensus.cc:2258] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.354734 27625 raft_consensus.cc:2229] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:48.354952 28466 ts_tablet_manager.cc:1875] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.354990 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:48.356854 28295 tablet_replica.cc:330] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:48.358597 24706 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:48.362437 28466 ts_tablet_manager.cc:1888] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.362498 28466 log.cc:1196] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/09d33aa5ebfd404ca218b069767d216b
I20240501 19:56:48.362746 28466 ts_tablet_manager.cc:1909] T 09d33aa5ebfd404ca218b069767d216b P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:48.362990 28780 tablet_replica.cc:330] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:48.363169 28780 raft_consensus.cc:2229] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.363282 28780 raft_consensus.cc:2258] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.363889 28780 ts_tablet_manager.cc:1875] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.364431 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 00b49c7daa8e49e9b264fdcab76ea976 (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.364738 27624 tablet_replica.cc:330] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:48.364873 27624 raft_consensus.cc:2229] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.365024 27624 raft_consensus.cc:2258] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.365522 27624 ts_tablet_manager.cc:1875] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.365584 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:48.367604 28780 ts_tablet_manager.cc:1888] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.367746 28780 log.cc:1196] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/f6dbe23d20b34a348938dcc3571020b7
I20240501 19:56:48.368028 27624 ts_tablet_manager.cc:1888] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.368103 27624 log.cc:1196] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/f6dbe23d20b34a348938dcc3571020b7
I20240501 19:56:48.368120 28780 ts_tablet_manager.cc:1909] T f6dbe23d20b34a348938dcc3571020b7 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:48.369657 28710 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:48.369942 28710 tablet.cc:1620] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:48.371143 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet f6dbe23d20b34a348938dcc3571020b7 (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.377600 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=a0c7f223593a4187bdb5f36c9339d403] alter complete (version 2)
I20240501 19:56:48.377676 28715 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:48.377957 28715 tablet.cc:1620] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:48.378931 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6b9bbaf2a0470482f3c5ff98322b90 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.379209 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6b9bbaf2a0470482f3c5ff98322b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.380434 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6b9bbaf2a0470482f3c5ff98322b90 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.380730 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6b9bbaf2a0470482f3c5ff98322b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.380848 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 166bb900935040bd9f7723e822780e6c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.381106 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166bb900935040bd9f7723e822780e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.381139 28559 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:48.381976 28559 tablet_bootstrap.cc:654] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.382498 28779 ts_tablet_manager.cc:1888] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.384083 28784 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:48.384145 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 14decb125f34452a81547e25a09fb966 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.384439 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14decb125f34452a81547e25a09fb966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.383307 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 14decb125f34452a81547e25a09fb966 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.385188 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14decb125f34452a81547e25a09fb966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.385493 28784 tablet_bootstrap.cc:654] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.387077 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 166bb900935040bd9f7723e822780e6c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.387334 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166bb900935040bd9f7723e822780e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.383540 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6b9bbaf2a0470482f3c5ff98322b90 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.387521 28784 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:48.387604 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6b9bbaf2a0470482f3c5ff98322b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.387619 28784 ts_tablet_manager.cc:1367] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:48.388087 28784 raft_consensus.cc:348] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.388222 28784 raft_consensus.cc:374] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.388267 28784 raft_consensus.cc:729] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.388413 28784 consensus_queue.cc:260] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.388739 28784 ts_tablet_manager.cc:1398] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:48.382581 28779 log.cc:1196] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/09d33aa5ebfd404ca218b069767d216b
I20240501 19:56:48.390192 28779 ts_tablet_manager.cc:1909] T 09d33aa5ebfd404ca218b069767d216b P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:48.396139 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 09d33aa5ebfd404ca218b069767d216b (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.396142 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 14decb125f34452a81547e25a09fb966 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.396409 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14decb125f34452a81547e25a09fb966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.398551 28558 tablet_bootstrap.cc:492] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:48.399621 28558 tablet_bootstrap.cc:654] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.369877 28717 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:48.400487 28717 tablet.cc:1620] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:48.403414 28295 raft_consensus.cc:2229] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:48.403692 28295 raft_consensus.cc:2258] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.404209 28295 ts_tablet_manager.cc:1875] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.406854 28295 ts_tablet_manager.cc:1888] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.406931 28295 log.cc:1196] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/f6dbe23d20b34a348938dcc3571020b7
I20240501 19:56:48.407245 28295 ts_tablet_manager.cc:1909] T f6dbe23d20b34a348938dcc3571020b7 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:48.407792 28558 tablet_bootstrap.cc:492] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:48.407888 28558 ts_tablet_manager.cc:1367] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:48.407972 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet f6dbe23d20b34a348938dcc3571020b7 (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.408411 28558 raft_consensus.cc:348] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.408528 28558 raft_consensus.cc:374] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.408563 28558 raft_consensus.cc:729] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
W20240501 19:56:48.357993 24923 consensus_peers.cc:485] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:48.408691 28558 consensus_queue.cc:260] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.408859 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 09d33aa5ebfd404ca218b069767d216b (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
W20240501 19:56:48.408800 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.65.3:38893, user_credentials={real_user=slave}} blocked reactor thread for 50815.1us
I20240501 19:56:48.409001 28558 ts_tablet_manager.cc:1398] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:48.408993 27625 raft_consensus.cc:2258] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.409538 27625 ts_tablet_manager.cc:1875] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.410387 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 166bb900935040bd9f7723e822780e6c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:48.410604 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166bb900935040bd9f7723e822780e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.411756 27625 ts_tablet_manager.cc:1888] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.411841 27625 log.cc:1196] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/09d33aa5ebfd404ca218b069767d216b
I20240501 19:56:48.412122 27625 ts_tablet_manager.cc:1909] T 09d33aa5ebfd404ca218b069767d216b P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:48.412361 28789 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:48.413473 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 09d33aa5ebfd404ca218b069767d216b (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.413902 28789 tablet_bootstrap.cc:654] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.416656 24836 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:48.417017 27624 ts_tablet_manager.cc:1909] T f6dbe23d20b34a348938dcc3571020b7 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:48.417618 28789 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:48.417711 28789 ts_tablet_manager.cc:1367] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:48.418123 28789 raft_consensus.cc:348] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.418238 28789 raft_consensus.cc:374] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.418280 28789 raft_consensus.cc:729] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.418398 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet f6dbe23d20b34a348938dcc3571020b7 (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.418398 28789 consensus_queue.cc:260] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.418728 28789 ts_tablet_manager.cc:1398] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:48.419175 27624 tablet_replica.cc:330] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:48.419332 27624 raft_consensus.cc:2229] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.419476 27624 raft_consensus.cc:2258] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.419991 27624 ts_tablet_manager.cc:1875] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.423907 27624 ts_tablet_manager.cc:1888] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.423977 27624 log.cc:1196] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/09d33aa5ebfd404ca218b069767d216b
I20240501 19:56:48.424242 27624 ts_tablet_manager.cc:1909] T 09d33aa5ebfd404ca218b069767d216b P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:48.426160 28785 tablet_bootstrap.cc:492] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:48.427129 28783 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:48.427371 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 09d33aa5ebfd404ca218b069767d216b (table rebalance_test_table_12 [id=44d3fe858f054489b443502e10e226cc]) successfully deleted
I20240501 19:56:48.428468 28783 tablet_bootstrap.cc:654] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.428741 28787 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:48.429055 28559 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:48.429145 28559 ts_tablet_manager.cc:1367] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.000s
I20240501 19:56:48.429627 28559 raft_consensus.cc:348] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.429761 28559 raft_consensus.cc:374] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.429798 28559 raft_consensus.cc:729] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.429926 28559 consensus_queue.cc:260] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.430238 28559 ts_tablet_manager.cc:1398] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:48.430884 28787 tablet_bootstrap.cc:654] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.431932 28785 tablet_bootstrap.cc:654] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.437760 28783 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:48.437938 28783 ts_tablet_manager.cc:1367] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:48.438555 28783 raft_consensus.cc:348] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.438768 28783 raft_consensus.cc:374] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.438850 28783 raft_consensus.cc:729] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.439079 28783 consensus_queue.cc:260] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.439546 28787 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:48.439687 28787 ts_tablet_manager.cc:1367] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:48.440133 28787 raft_consensus.cc:348] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.441689 28787 raft_consensus.cc:374] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.441785 28787 raft_consensus.cc:729] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.441984 28787 consensus_queue.cc:260] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.442385 28787 ts_tablet_manager.cc:1398] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:48.442606 28787 tablet_bootstrap.cc:492] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:48.443562 28787 tablet_bootstrap.cc:654] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.443665 28334 raft_consensus.cc:482] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:48.443760 28334 raft_consensus.cc:504] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.443861 28783 ts_tablet_manager.cc:1398] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:48.443876 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:48.444167 28334 leader_election.cc:288] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:48.443588 27012 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:48.444965 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14decb125f34452a81547e25a09fb966" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
W20240501 19:56:48.445845 27227 leader_election.cc:341] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:48.453922 28785 tablet_bootstrap.cc:492] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:48.454113 28785 ts_tablet_manager.cc:1367] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240501 19:56:48.454566 28785 raft_consensus.cc:348] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.454730 28785 raft_consensus.cc:374] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.454773 28785 raft_consensus.cc:729] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.454931 28785 consensus_queue.cc:260] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.455350 28785 ts_tablet_manager.cc:1398] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:48.455916 28785 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:48.457005 28785 tablet_bootstrap.cc:654] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.460309 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14decb125f34452a81547e25a09fb966" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:48.460456 27425 raft_consensus.cc:2454] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:48.460721 27229 leader_election.cc:302] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:48.460909 28334 raft_consensus.cc:2781] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:48.460981 28334 raft_consensus.cc:482] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:48.461036 28334 raft_consensus.cc:3037] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.461876 28334 raft_consensus.cc:504] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.462251 28334 leader_election.cc:288] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:48.462821 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14decb125f34452a81547e25a09fb966" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:48.462878 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14decb125f34452a81547e25a09fb966" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:48.462975 27425 raft_consensus.cc:3037] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:56:48.463586 27227 leader_election.cc:341] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:48.463996 27425 raft_consensus.cc:2454] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:48.464269 27229 leader_election.cc:302] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:48.464435 28334 raft_consensus.cc:2781] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:48.464527 28334 raft_consensus.cc:686] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:48.464653 28334 consensus_queue.cc:237] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.466552 24446 catalog_manager.cc:5484] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:48.467463 28787 tablet_bootstrap.cc:492] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:48.467562 28787 ts_tablet_manager.cc:1367] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 19:56:48.468003 28787 raft_consensus.cc:348] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.468134 28787 raft_consensus.cc:374] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.468174 28787 raft_consensus.cc:729] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.468312 28787 consensus_queue.cc:260] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.468569 28787 ts_tablet_manager.cc:1398] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:48.469805 28785 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:48.470474 28785 ts_tablet_manager.cc:1367] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:56:48.471242 28785 raft_consensus.cc:348] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.471336 28785 raft_consensus.cc:374] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.471372 28785 raft_consensus.cc:729] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.471967 28785 consensus_queue.cc:260] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.473202 28785 ts_tablet_manager.cc:1398] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:48.508915 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:48.514384 28710 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:48.514760 28717 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:48.520781 28715 tablet.cc:1722] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:48.521342 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=a0c7f223593a4187bdb5f36c9339d403] alter complete (version 3)
I20240501 19:56:48.543948 26677 raft_consensus.cc:482] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:48.544073 26677 raft_consensus.cc:504] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.544602 26677 leader_election.cc:288] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:48.545500 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6b9bbaf2a0470482f3c5ff98322b90" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:48.545680 27031 raft_consensus.cc:2454] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:48.546182 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6b9bbaf2a0470482f3c5ff98322b90" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:48.546303 24856 raft_consensus.cc:2454] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:48.546579 24661 leader_election.cc:302] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:48.546808 26677 raft_consensus.cc:2781] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:48.546875 26677 raft_consensus.cc:482] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:48.546931 26677 raft_consensus.cc:3037] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.548027 26677 raft_consensus.cc:504] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.548379 26677 leader_election.cc:288] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:48.548812 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6b9bbaf2a0470482f3c5ff98322b90" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:48.548940 27031 raft_consensus.cc:3037] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.549827 27031 raft_consensus.cc:2454] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:48.550097 24660 leader_election.cc:302] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:48.550135 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6b9bbaf2a0470482f3c5ff98322b90" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:48.550262 26677 raft_consensus.cc:2781] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:48.550251 24856 raft_consensus.cc:3037] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.550339 26677 raft_consensus.cc:686] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:48.550495 26677 consensus_queue.cc:237] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:48.551019 24856 raft_consensus.cc:2454] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:48.552549 24452 catalog_manager.cc:5484] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:48.625437 24449 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:48.625573 24449 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:48.627619 24449 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 03f6fc63ecb1457486fbabacab60d2fb
I20240501 19:56:48.628319 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f6fc63ecb1457486fbabacab60d2fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:48.628520 27624 tablet_replica.cc:330] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:48.628489 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f6fc63ecb1457486fbabacab60d2fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:56:48.628666 27624 raft_consensus.cc:2229] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:48.628840 28839 tablet_replica.cc:330] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:56:48.628890 27624 raft_consensus.cc:2258] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.628990 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f6fc63ecb1457486fbabacab60d2fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:48.629009 28839 raft_consensus.cc:2229] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.629124 28839 raft_consensus.cc:2258] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.629191 28295 tablet_replica.cc:330] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:48.629314 28295 raft_consensus.cc:2229] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.629441 28295 raft_consensus.cc:2258] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.629577 27624 ts_tablet_manager.cc:1875] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.629688 28839 ts_tablet_manager.cc:1875] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.630329 28295 ts_tablet_manager.cc:1875] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.631922 27624 ts_tablet_manager.cc:1888] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.631970 28839 ts_tablet_manager.cc:1888] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.631989 27624 log.cc:1196] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/03f6fc63ecb1457486fbabacab60d2fb
I20240501 19:56:48.632030 28839 log.cc:1196] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/03f6fc63ecb1457486fbabacab60d2fb
I20240501 19:56:48.632251 28295 ts_tablet_manager.cc:1888] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.632272 27624 ts_tablet_manager.cc:1909] T 03f6fc63ecb1457486fbabacab60d2fb P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:48.632300 28295 log.cc:1196] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/03f6fc63ecb1457486fbabacab60d2fb
I20240501 19:56:48.632308 28839 ts_tablet_manager.cc:1909] T 03f6fc63ecb1457486fbabacab60d2fb P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:56:48.632544 28295 ts_tablet_manager.cc:1909] T 03f6fc63ecb1457486fbabacab60d2fb P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:48.633150 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 03f6fc63ecb1457486fbabacab60d2fb (table rebalancer_dynamic_table_new_name [id=a0c7f223593a4187bdb5f36c9339d403]) successfully deleted
I20240501 19:56:48.633230 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 03f6fc63ecb1457486fbabacab60d2fb (table rebalancer_dynamic_table_new_name [id=a0c7f223593a4187bdb5f36c9339d403]) successfully deleted
I20240501 19:56:48.633867 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 03f6fc63ecb1457486fbabacab60d2fb (table rebalancer_dynamic_table_new_name [id=a0c7f223593a4187bdb5f36c9339d403]) successfully deleted
I20240501 19:56:48.729254 24449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:48.734867 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 794ebea8629f4300b4ce1ce77e45ea81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e7c9331c261645acaa397816ae79f12a]), partition=
I20240501 19:56:48.734983 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 794ebea8629f4300b4ce1ce77e45ea81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e7c9331c261645acaa397816ae79f12a]), partition=
I20240501 19:56:48.734998 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 794ebea8629f4300b4ce1ce77e45ea81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e7c9331c261645acaa397816ae79f12a]), partition=
I20240501 19:56:48.735193 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794ebea8629f4300b4ce1ce77e45ea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.735256 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794ebea8629f4300b4ce1ce77e45ea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.735291 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794ebea8629f4300b4ce1ce77e45ea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.736966 28785 tablet_bootstrap.cc:492] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:48.737227 28558 tablet_bootstrap.cc:492] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:48.737268 28784 tablet_bootstrap.cc:492] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:48.737830 28785 tablet_bootstrap.cc:654] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.738046 28784 tablet_bootstrap.cc:654] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.738078 28558 tablet_bootstrap.cc:654] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.738973 28785 tablet_bootstrap.cc:492] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:48.739092 28785 ts_tablet_manager.cc:1367] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:48.739547 28785 raft_consensus.cc:348] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.739674 28785 raft_consensus.cc:374] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.739708 28785 raft_consensus.cc:729] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.739723 28558 tablet_bootstrap.cc:492] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:48.739792 28558 ts_tablet_manager.cc:1367] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:48.739830 28785 consensus_queue.cc:260] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.740084 28785 ts_tablet_manager.cc:1398] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:48.740181 28558 raft_consensus.cc:348] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.740283 28558 raft_consensus.cc:374] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.740319 28558 raft_consensus.cc:729] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.740433 28558 consensus_queue.cc:260] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.740725 28558 ts_tablet_manager.cc:1398] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:48.742101 28784 tablet_bootstrap.cc:492] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:48.742193 28784 ts_tablet_manager.cc:1367] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:48.742662 28784 raft_consensus.cc:348] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.742774 28784 raft_consensus.cc:374] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.742826 28784 raft_consensus.cc:729] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.742955 28784 consensus_queue.cc:260] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.743230 28784 ts_tablet_manager.cc:1398] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:48.776666 28843 raft_consensus.cc:482] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:48.776793 28843 raft_consensus.cc:504] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.777185 28843 leader_election.cc:288] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:48.777561 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166bb900935040bd9f7723e822780e6c" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:48.777702 27425 raft_consensus.cc:2454] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:48.777714 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166bb900935040bd9f7723e822780e6c" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:48.777868 27162 raft_consensus.cc:2454] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:48.778048 24924 leader_election.cc:302] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:48.778209 28843 raft_consensus.cc:2781] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:48.778270 28843 raft_consensus.cc:482] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:48.778318 28843 raft_consensus.cc:3037] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.779314 28843 raft_consensus.cc:504] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.779625 28843 leader_election.cc:288] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:48.779946 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166bb900935040bd9f7723e822780e6c" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:48.780043 27425 raft_consensus.cc:3037] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.780082 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166bb900935040bd9f7723e822780e6c" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:48.780234 27162 raft_consensus.cc:3037] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.780910 27425 raft_consensus.cc:2454] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:48.781019 27162 raft_consensus.cc:2454] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:48.781256 24924 leader_election.cc:302] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:48.781432 28843 raft_consensus.cc:2781] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:48.781539 28843 raft_consensus.cc:686] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:48.781670 28843 consensus_queue.cc:237] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:48.783481 24452 catalog_manager.cc:5484] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:48.783775 28542 raft_consensus.cc:482] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:48.783866 28542 raft_consensus.cc:504] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.784240 28542 leader_election.cc:288] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:48.784607 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14decb125f34452a81547e25a09fb966" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:48.784917 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14decb125f34452a81547e25a09fb966" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:48.785068 27425 raft_consensus.cc:2379] T 14decb125f34452a81547e25a09fb966 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in current term 1: Already voted for candidate 53e34bb09d8e41129a5683079ff63610 in this term.
I20240501 19:56:48.785353 26967 leader_election.cc:302] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:48.785547 28542 raft_consensus.cc:3037] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.786567 28542 raft_consensus.cc:2726] T 14decb125f34452a81547e25a09fb966 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:48.806268 24836 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:48.807173 27012 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:48.808501 24706 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:48.808997 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:48.810086 27405 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:48.815505 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:48.816624 27143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:48.839372 28518 raft_consensus.cc:482] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:48.839511 28518 raft_consensus.cc:504] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.839956 28518 leader_election.cc:288] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:48.841504 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ebea8629f4300b4ce1ce77e45ea81" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:48.841509 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ebea8629f4300b4ce1ce77e45ea81" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:48.841655 27425 raft_consensus.cc:2454] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:48.841665 27276 raft_consensus.cc:2454] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:48.842624 27098 leader_election.cc:302] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:48.844782 28518 raft_consensus.cc:2781] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:48.844849 28518 raft_consensus.cc:482] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:48.844902 28518 raft_consensus.cc:3037] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.846041 28518 raft_consensus.cc:504] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.846400 28518 leader_election.cc:288] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:48.847126 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ebea8629f4300b4ce1ce77e45ea81" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:48.847254 27276 raft_consensus.cc:3037] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.848151 27276 raft_consensus.cc:2454] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:48.848501 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ebea8629f4300b4ce1ce77e45ea81" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:48.848635 27425 raft_consensus.cc:3037] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.848767 27098 leader_election.cc:302] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:48.849057 28518 raft_consensus.cc:2781] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:48.849205 28518 raft_consensus.cc:686] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:48.849334 28518 consensus_queue.cc:237] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:48.849685 27425 raft_consensus.cc:2454] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:48.850216 24448 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 19:56:48.850340 24448 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 19:56:48.851135 24452 catalog_manager.cc:5484] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:48.859812 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 4 replicas of tablet 6f1c61cca30c4d878d921c15158a5947
I20240501 19:56:48.860136 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 4 replicas of tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c
I20240501 19:56:48.860342 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet fe354fb491da4c8d896e46b5e1f6edf2
I20240501 19:56:48.860615 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:48.860636 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:48.860774 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c61cca30c4d878d921c15158a5947 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:48.860807 28466 tablet_replica.cc:330] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:48.860965 28466 raft_consensus.cc:2229] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:48.861135 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c61cca30c4d878d921c15158a5947 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:48.861179 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:48.861223 28466 raft_consensus.cc:2258] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.861259 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet fe354fb491da4c8d896e46b5e1f6edf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:48.861325 27624 tablet_replica.cc:330] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:48.861421 27624 raft_consensus.cc:2229] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.861544 27624 raft_consensus.cc:2258] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.861577 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c61cca30c4d878d921c15158a5947 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:48.861726 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet fe354fb491da4c8d896e46b5e1f6edf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:48.861830 28466 ts_tablet_manager.cc:1875] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.862038 27624 ts_tablet_manager.cc:1875] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.863415 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:48.864181 28858 tablet_replica.cc:330] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:48.864200 28466 ts_tablet_manager.cc:1888] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.864282 27624 ts_tablet_manager.cc:1888] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.864346 27624 log.cc:1196] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f1c61cca30c4d878d921c15158a5947
I20240501 19:56:48.864362 27625 tablet_replica.cc:330] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:48.864380 28858 raft_consensus.cc:2229] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.864498 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:48.864513 27625 raft_consensus.cc:2229] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.864548 28858 raft_consensus.cc:2258] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.864603 27624 ts_tablet_manager.cc:1909] T 6f1c61cca30c4d878d921c15158a5947 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:48.864687 27625 raft_consensus.cc:2258] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.864878 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet fe354fb491da4c8d896e46b5e1f6edf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:48.865213 27625 ts_tablet_manager.cc:1875] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.865262 28858 ts_tablet_manager.cc:1875] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.865577 27624 tablet_replica.cc:330] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:48.865693 27624 raft_consensus.cc:2229] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.865819 27624 raft_consensus.cc:2258] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.866271 27624 ts_tablet_manager.cc:1875] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.866376 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 6f1c61cca30c4d878d921c15158a5947 (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.867144 28295 tablet_replica.cc:330] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:48.867303 28295 raft_consensus.cc:2229] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:48.867561 28295 raft_consensus.cc:2258] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.868038 28858 ts_tablet_manager.cc:1888] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.868044 28295 ts_tablet_manager.cc:1875] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.868103 28858 log.cc:1196] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/3f2d49dcdd8f49d48d8a9bd6d10c332c
I20240501 19:56:48.868417 27624 ts_tablet_manager.cc:1888] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.868474 27624 log.cc:1196] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/3f2d49dcdd8f49d48d8a9bd6d10c332c
I20240501 19:56:48.868376 28858 ts_tablet_manager.cc:1909] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:48.868837 27624 ts_tablet_manager.cc:1909] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:48.869223 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.869309 27624 tablet_replica.cc:330] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:48.869391 27624 raft_consensus.cc:2229] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:48.869516 27624 raft_consensus.cc:2258] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.869495 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.869905 27624 ts_tablet_manager.cc:1875] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.870126 28295 ts_tablet_manager.cc:1888] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.870173 28295 log.cc:1196] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f1c61cca30c4d878d921c15158a5947
I20240501 19:56:48.870355 28295 ts_tablet_manager.cc:1909] T 6f1c61cca30c4d878d921c15158a5947 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:48.870529 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1c61cca30c4d878d921c15158a5947 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:48 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:48.870714 28779 tablet_replica.cc:330] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:48.870831 28779 raft_consensus.cc:2229] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.870923 28779 raft_consensus.cc:2258] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.871158 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 6f1c61cca30c4d878d921c15158a5947 (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.871348 28295 tablet_replica.cc:330] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:48.871461 28295 raft_consensus.cc:2229] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.871539 28779 ts_tablet_manager.cc:1875] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.871547 27624 ts_tablet_manager.cc:1888] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:48.871588 28295 raft_consensus.cc:2258] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.871604 27624 log.cc:1196] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe354fb491da4c8d896e46b5e1f6edf2
I20240501 19:56:48.871848 27624 ts_tablet_manager.cc:1909] T fe354fb491da4c8d896e46b5e1f6edf2 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:48.872071 28295 ts_tablet_manager.cc:1875] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.872406 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet fe354fb491da4c8d896e46b5e1f6edf2 (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.873631 28779 ts_tablet_manager.cc:1888] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.873695 28779 log.cc:1196] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/6f1c61cca30c4d878d921c15158a5947
I20240501 19:56:48.874243 28295 ts_tablet_manager.cc:1888] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:48.874297 28295 log.cc:1196] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe354fb491da4c8d896e46b5e1f6edf2
I20240501 19:56:48.874629 28295 ts_tablet_manager.cc:1909] T fe354fb491da4c8d896e46b5e1f6edf2 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:48.874667 28779 ts_tablet_manager.cc:1909] T 6f1c61cca30c4d878d921c15158a5947 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:48.875327 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet fe354fb491da4c8d896e46b5e1f6edf2 (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.876523 27625 ts_tablet_manager.cc:1888] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.876595 27625 log.cc:1196] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/3f2d49dcdd8f49d48d8a9bd6d10c332c
I20240501 19:56:48.877015 27625 ts_tablet_manager.cc:1909] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:48.877514 27625 tablet_replica.cc:330] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:48.877640 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.877643 28335 consensus_queue.cc:1035] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.878158 28466 log.cc:1196] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f2d49dcdd8f49d48d8a9bd6d10c332c
I20240501 19:56:48.878809 28466 ts_tablet_manager.cc:1909] T 3f2d49dcdd8f49d48d8a9bd6d10c332c P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:48.879407 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 6f1c61cca30c4d878d921c15158a5947 (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.879907 27625 raft_consensus.cc:2229] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.880679 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.883684 28334 consensus_queue.cc:1035] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.884837 27625 raft_consensus.cc:2258] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.885382 27625 ts_tablet_manager.cc:1875] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.887882 27625 ts_tablet_manager.cc:1888] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:48.887945 27625 log.cc:1196] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f1c61cca30c4d878d921c15158a5947
I20240501 19:56:48.888206 27625 ts_tablet_manager.cc:1909] T 6f1c61cca30c4d878d921c15158a5947 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:48.889883 27625 tablet_replica.cc:330] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:48.889909 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 6f1c61cca30c4d878d921c15158a5947 (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.890039 27625 raft_consensus.cc:2229] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.890156 27625 raft_consensus.cc:2258] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.890630 27625 ts_tablet_manager.cc:1875] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:48.892395 27625 ts_tablet_manager.cc:1888] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:48.892472 27625 log.cc:1196] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/fe354fb491da4c8d896e46b5e1f6edf2
I20240501 19:56:48.892701 27625 ts_tablet_manager.cc:1909] T fe354fb491da4c8d896e46b5e1f6edf2 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:48.893378 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet fe354fb491da4c8d896e46b5e1f6edf2 (table rebalance_test_table_14 [id=2eb305082d57496294609737a2b6cb47]) successfully deleted
I20240501 19:56:48.916662 24727 consensus_queue.cc:237] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:48.917771 27031 raft_consensus.cc:1261] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:48.917793 24856 raft_consensus.cc:1261] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:48.918140 27637 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:48.918282 26677 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:48.920132 28709 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:48.920635 24856 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:48.922122 27031 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:48.923955 24448 catalog_manager.cc:5484] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:48.926990 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0bb0d3396a436c878f56d157f30eda with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:48.927263 28295 tablet_replica.cc:330] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:48.927405 28295 raft_consensus.cc:2229] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:48.927556 28295 raft_consensus.cc:2258] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:48.928169 28295 ts_tablet_manager.cc:1875] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:48.930241 28295 ts_tablet_manager.cc:1888] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:48.930298 28295 log.cc:1196] T cb0bb0d3396a436c878f56d157f30eda P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/cb0bb0d3396a436c878f56d157f30eda
I20240501 19:56:48.932566 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet cb0bb0d3396a436c878f56d157f30eda (table rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]) successfully deleted
I20240501 19:56:48.969360 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:48.976613 27425 raft_consensus.cc:1261] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:48.977042 27276 raft_consensus.cc:1261] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:48.977200 28518 consensus_queue.cc:1035] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.977459 28518 consensus_queue.cc:1035] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:48.979717 24451 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 794ebea8629f4300b4ce1ce77e45ea81 (table rebalancer_dynamic_table [id=e7c9331c261645acaa397816ae79f12a]). Expected version 1 got 0
I20240501 19:56:48.980444 24448 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet 794ebea8629f4300b4ce1ce77e45ea81 (table rebalancer_dynamic_table [id=e7c9331c261645acaa397816ae79f12a]). Expected version 1 got 0
I20240501 19:56:48.980610 28923 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:48.980646 28862 mvcc.cc:205] Tried to move back new op lower bound from 7022974603161796608 to 7022974602648440832. Current Snapshot: MvccSnapshot[applied={T|T < 7022974603161796608}]
I20240501 19:56:48.980913 28923 tablet.cc:1620] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:48.981822 28924 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:48.982188 28924 tablet.cc:1620] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:48.982743 24451 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 794ebea8629f4300b4ce1ce77e45ea81 (table rebalancer_dynamic_table [id=e7c9331c261645acaa397816ae79f12a]). Expected version 1 got 0
I20240501 19:56:48.983963 28715 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:48.984246 28715 tablet.cc:1620] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:48.987229 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=e7c9331c261645acaa397816ae79f12a] alter complete (version 1)
I20240501 19:56:49.014390 26677 consensus_queue.cc:1035] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.016197 27405 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:49.018884 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:49.019984 27143 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:49.024729 27012 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:49.031628 26677 consensus_queue.cc:1035] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.034371 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:49.034533 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.059882 24968 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:49.069590 24576 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:49.088958 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:49.094698 28923 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:49.095026 28923 tablet.cc:1620] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:49.095160 28715 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:49.095321 28924 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:49.095428 28715 tablet.cc:1620] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:49.095608 28924 tablet.cc:1620] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:49.099659 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=e7c9331c261645acaa397816ae79f12a] alter complete (version 2)
I20240501 19:56:49.112179 24727 consensus_queue.cc:237] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:49.113312 27276 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:49.113317 24856 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:49.113682 28709 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:49.114089 28709 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:49.115269 26677 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.116080 24856 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.116638 27276 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.119194 24452 catalog_manager.cc:5484] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.121722 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfa6701a958416ca481e0349071caa0 with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.121994 28295 tablet_replica.cc:330] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.122139 28295 raft_consensus.cc:2229] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.122285 28295 raft_consensus.cc:2258] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.122764 28295 ts_tablet_manager.cc:1875] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:49.124568 28295 ts_tablet_manager.cc:1888] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:49.124666 28295 log.cc:1196] T fcfa6701a958416ca481e0349071caa0 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/fcfa6701a958416ca481e0349071caa0
I20240501 19:56:49.125167 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet fcfa6701a958416ca481e0349071caa0 (table rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]) successfully deleted
I20240501 19:56:49.202072 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:49.202275 27012 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:49.205643 27143 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:49.212460 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.218152 28923 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:49.220336 24576 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:49.222779 28924 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:49.223404 28715 tablet.cc:1722] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:49.230360 27405 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:49.230409 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=e7c9331c261645acaa397816ae79f12a] alter complete (version 3)
I20240501 19:56:49.233327 28843 consensus_queue.cc:1035] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.235121 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:49.239158 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:49.239192 24968 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:49.253331 28843 consensus_queue.cc:1035] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.298643 24987 consensus_queue.cc:237] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:49.299902 24727 raft_consensus.cc:1261] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:49.299945 27276 raft_consensus.cc:1261] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:49.300274 28843 consensus_queue.cc:1035] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:49.300470 28694 consensus_queue.cc:1035] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:49.302760 28843 raft_consensus.cc:2932] T 66215de525314ea180c164667f4e9f3f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.303097 24727 raft_consensus.cc:2932] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.303187 27276 raft_consensus.cc:2932] T 66215de525314ea180c164667f4e9f3f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.305931 24447 catalog_manager.cc:5484] T 66215de525314ea180c164667f4e9f3f P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.308825 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 66215de525314ea180c164667f4e9f3f with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:49.309047 28466 tablet_replica.cc:330] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:49.309185 28466 raft_consensus.cc:2229] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.309293 28466 raft_consensus.cc:2258] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.309738 28466 ts_tablet_manager.cc:1875] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:49.311434 28466 ts_tablet_manager.cc:1888] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:49.311508 28466 log.cc:1196] T 66215de525314ea180c164667f4e9f3f P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/66215de525314ea180c164667f4e9f3f
I20240501 19:56:49.312007 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 66215de525314ea180c164667f4e9f3f (table rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]) successfully deleted
I20240501 19:56:49.337185 24453 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:49.337317 24453 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:49.339469 24453 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 794ebea8629f4300b4ce1ce77e45ea81
I20240501 19:56:49.340137 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet 794ebea8629f4300b4ce1ce77e45ea81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:56:49.340320 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 794ebea8629f4300b4ce1ce77e45ea81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:49.340584 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 794ebea8629f4300b4ce1ce77e45ea81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:49.340777 28858 tablet_replica.cc:330] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:49.341313 28858 raft_consensus.cc:2229] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.341475 28858 raft_consensus.cc:2258] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.341722 29048 tablet_replica.cc:330] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:49.341723 29047 tablet_replica.cc:330] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:56:49.341856 29047 raft_consensus.cc:2229] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.341915 29048 raft_consensus.cc:2229] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:49.342020 29047 raft_consensus.cc:2258] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.342132 29048 raft_consensus.cc:2258] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.342177 28858 ts_tablet_manager.cc:1875] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.342586 29047 ts_tablet_manager.cc:1875] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.342695 29048 ts_tablet_manager.cc:1875] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.344132 28858 ts_tablet_manager.cc:1888] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.344218 28858 log.cc:1196] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/794ebea8629f4300b4ce1ce77e45ea81
I20240501 19:56:49.344483 28858 ts_tablet_manager.cc:1909] T 794ebea8629f4300b4ce1ce77e45ea81 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:49.344691 29048 ts_tablet_manager.cc:1888] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.344754 29048 log.cc:1196] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/794ebea8629f4300b4ce1ce77e45ea81
I20240501 19:56:49.344676 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:49.345046 29048 ts_tablet_manager.cc:1909] T 794ebea8629f4300b4ce1ce77e45ea81 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:49.346082 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 794ebea8629f4300b4ce1ce77e45ea81 (table rebalancer_dynamic_table_new_name [id=e7c9331c261645acaa397816ae79f12a]) successfully deleted
I20240501 19:56:49.346959 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 794ebea8629f4300b4ce1ce77e45ea81 (table rebalancer_dynamic_table_new_name [id=e7c9331c261645acaa397816ae79f12a]) successfully deleted
I20240501 19:56:49.347522 29047 ts_tablet_manager.cc:1888] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.347592 29047 log.cc:1196] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/794ebea8629f4300b4ce1ce77e45ea81
I20240501 19:56:49.347882 29047 ts_tablet_manager.cc:1909] T 794ebea8629f4300b4ce1ce77e45ea81 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:56:49.348524 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 794ebea8629f4300b4ce1ce77e45ea81 (table rebalancer_dynamic_table_new_name [id=e7c9331c261645acaa397816ae79f12a]) successfully deleted
I20240501 19:56:49.350414 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 36d90e332364430d980a08be1f8a5c3a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.350739 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d90e332364430d980a08be1f8a5c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.351276 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 8b747487a083433d91191aa79458f51e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.351356 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 36d90e332364430d980a08be1f8a5c3a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.351548 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b747487a083433d91191aa79458f51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.352078 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d90e332364430d980a08be1f8a5c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.352155 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 36d90e332364430d980a08be1f8a5c3a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.352413 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d90e332364430d980a08be1f8a5c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.353698 29055 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:49.355176 29055 tablet_bootstrap.cc:654] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.355216 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4c18dfded64901bb4645af3dbe2dbc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.355249 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4c18dfded64901bb4645af3dbe2dbc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.355341 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4c18dfded64901bb4645af3dbe2dbc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.355490 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4c18dfded64901bb4645af3dbe2dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.355506 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4c18dfded64901bb4645af3dbe2dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.355552 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4c18dfded64901bb4645af3dbe2dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.356225 29055 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:49.356326 29055 ts_tablet_manager.cc:1367] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:49.356829 29055 raft_consensus.cc:348] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.356920 29055 raft_consensus.cc:374] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.356946 29055 raft_consensus.cc:729] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.357053 29055 consensus_queue.cc:260] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.357343 29055 ts_tablet_manager.cc:1398] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:49.357522 29055 tablet_bootstrap.cc:492] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:49.357666 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 8b747487a083433d91191aa79458f51e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.357826 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b747487a083433d91191aa79458f51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.358299 29055 tablet_bootstrap.cc:654] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.359134 29060 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:49.359386 29055 tablet_bootstrap.cc:492] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:49.359477 29055 ts_tablet_manager.cc:1367] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:49.359647 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 8b747487a083433d91191aa79458f51e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:49.359915 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b747487a083433d91191aa79458f51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.359889 29055 raft_consensus.cc:348] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.359977 29055 raft_consensus.cc:374] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.360004 29055 raft_consensus.cc:729] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.360116 29055 consensus_queue.cc:260] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.360383 29055 ts_tablet_manager.cc:1398] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:49.360968 29060 tablet_bootstrap.cc:654] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.361840 29056 tablet_bootstrap.cc:492] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:49.362783 29058 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:49.363265 29056 tablet_bootstrap.cc:654] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.363831 29059 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:49.364308 29060 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:49.364403 29060 ts_tablet_manager.cc:1367] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:49.365093 29060 raft_consensus.cc:348] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.365267 29060 raft_consensus.cc:374] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.365347 29060 raft_consensus.cc:729] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.365403 29059 tablet_bootstrap.cc:654] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.365906 29056 tablet_bootstrap.cc:492] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:49.366068 29058 tablet_bootstrap.cc:654] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.366040 29060 consensus_queue.cc:260] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.366930 29054 tablet_bootstrap.cc:492] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:49.366983 29060 ts_tablet_manager.cc:1398] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:49.367249 29056 ts_tablet_manager.cc:1367] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:49.367769 29056 raft_consensus.cc:348] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.367923 29056 raft_consensus.cc:374] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.368046 29056 raft_consensus.cc:729] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.368095 29054 tablet_bootstrap.cc:654] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.368224 29056 consensus_queue.cc:260] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.369248 29056 ts_tablet_manager.cc:1398] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:49.371477 29059 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:49.371567 29059 ts_tablet_manager.cc:1367] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:49.372023 29059 raft_consensus.cc:348] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.372118 29059 raft_consensus.cc:374] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.372148 29059 raft_consensus.cc:729] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.372277 29059 consensus_queue.cc:260] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.372875 29058 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:49.372960 29058 ts_tablet_manager.cc:1367] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:49.373399 29058 raft_consensus.cc:348] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.373488 29058 raft_consensus.cc:374] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.373514 29058 raft_consensus.cc:729] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.374945 29058 consensus_queue.cc:260] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.375653 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 19:56:49.375826 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 19:56:49.378295 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet a40a737a78824728b9e85e6a4ded638a
I20240501 19:56:49.378612 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 4 replicas of tablet d56fe9e75a9c418f817185542ae4c1d9
I20240501 19:56:49.378862 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 4 replicas of tablet e1141ea148d74ed594effc258ad44dd1
I20240501 19:56:49.379380 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet a40a737a78824728b9e85e6a4ded638a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.379510 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet d56fe9e75a9c418f817185542ae4c1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.379572 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet a40a737a78824728b9e85e6a4ded638a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:49.379652 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet d56fe9e75a9c418f817185542ae4c1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:49.379742 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet e1141ea148d74ed594effc258ad44dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.379765 28466 tablet_replica.cc:330] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:49.379824 28295 tablet_replica.cc:330] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.379907 28466 raft_consensus.cc:2229] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.379945 28295 raft_consensus.cc:2229] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.380045 28466 raft_consensus.cc:2258] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.380076 28295 raft_consensus.cc:2258] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.380318 29058 ts_tablet_manager.cc:1398] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:56:49.380450 29058 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:49.380540 28466 ts_tablet_manager.cc:1875] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.380540 28295 ts_tablet_manager.cc:1875] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.381145 29058 tablet_bootstrap.cc:654] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.382292 29059 ts_tablet_manager.cc:1398] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:56:49.382429 29059 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:49.382640 28295 ts_tablet_manager.cc:1888] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.382640 28466 ts_tablet_manager.cc:1888] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.382704 28295 log.cc:1196] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/a40a737a78824728b9e85e6a4ded638a
I20240501 19:56:49.382716 28466 log.cc:1196] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/a40a737a78824728b9e85e6a4ded638a
I20240501 19:56:49.383002 28466 ts_tablet_manager.cc:1909] T a40a737a78824728b9e85e6a4ded638a P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:49.383002 28295 ts_tablet_manager.cc:1909] T a40a737a78824728b9e85e6a4ded638a P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:49.383180 27625 tablet_replica.cc:330] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:49.383180 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet d56fe9e75a9c418f817185542ae4c1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:49.383214 29059 tablet_bootstrap.cc:654] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.383327 27625 raft_consensus.cc:2229] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.383489 27625 raft_consensus.cc:2258] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.384040 27625 ts_tablet_manager.cc:1875] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.384245 29059 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:49.384335 29059 ts_tablet_manager.cc:1367] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:49.384384 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet e1141ea148d74ed594effc258ad44dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:56:49.384774 29059 raft_consensus.cc:348] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.384883 29059 raft_consensus.cc:374] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.384913 29059 raft_consensus.cc:729] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.385015 29059 consensus_queue.cc:260] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.385242 29059 ts_tablet_manager.cc:1398] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:49.385269 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet a40a737a78824728b9e85e6a4ded638a (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.385399 29047 tablet_replica.cc:330] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:56:49.385516 29047 raft_consensus.cc:2229] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.385517 28295 tablet_replica.cc:330] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.385607 29047 raft_consensus.cc:2258] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.385641 28295 raft_consensus.cc:2229] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.385777 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet e1141ea148d74ed594effc258ad44dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:49.385962 28295 raft_consensus.cc:2258] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.386301 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet e1141ea148d74ed594effc258ad44dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:49.386453 29047 ts_tablet_manager.cc:1875] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.388036 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet d56fe9e75a9c418f817185542ae4c1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:49.388592 29058 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:49.388783 29058 ts_tablet_manager.cc:1367] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
W20240501 19:56:49.389518 24661 consensus_peers.cc:485] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.389676 29058 raft_consensus.cc:348] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.389839 29058 raft_consensus.cc:374] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.389869 29058 raft_consensus.cc:729] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.390390 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet a40a737a78824728b9e85e6a4ded638a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:49.390373 29058 consensus_queue.cc:260] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.390745 29058 ts_tablet_manager.cc:1398] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:49.391109 28295 ts_tablet_manager.cc:1875] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.391247 29047 ts_tablet_manager.cc:1888] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.391350 29047 log.cc:1196] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/e1141ea148d74ed594effc258ad44dd1
I20240501 19:56:49.391669 29047 ts_tablet_manager.cc:1909] T e1141ea148d74ed594effc258ad44dd1 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:56:49.391806 27625 ts_tablet_manager.cc:1888] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.391886 27625 log.cc:1196] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/d56fe9e75a9c418f817185542ae4c1d9
I20240501 19:56:49.392165 27625 ts_tablet_manager.cc:1909] T d56fe9e75a9c418f817185542ae4c1d9 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:49.392364 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet e1141ea148d74ed594effc258ad44dd1 (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.392742 26677 raft_consensus.cc:482] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:49.392844 26677 raft_consensus.cc:504] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.393293 26677 leader_election.cc:288] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:49.393582 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d90e332364430d980a08be1f8a5c3a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:49.393621 28295 ts_tablet_manager.cc:1888] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.393679 28295 log.cc:1196] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/d56fe9e75a9c418f817185542ae4c1d9
I20240501 19:56:49.393923 28295 ts_tablet_manager.cc:1909] T d56fe9e75a9c418f817185542ae4c1d9 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
W20240501 19:56:49.393915 24660 leader_election.cc:341] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:49.394485 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet d56fe9e75a9c418f817185542ae4c1d9 (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.394665 28295 tablet_replica.cc:330] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.394764 28295 raft_consensus.cc:2229] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.394886 28295 raft_consensus.cc:2258] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.395320 28295 ts_tablet_manager.cc:1875] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.396528 28466 tablet_replica.cc:330] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:49.396648 28466 raft_consensus.cc:2229] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:49.396844 28466 raft_consensus.cc:2258] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.397274 28466 ts_tablet_manager.cc:1875] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.398162 28295 ts_tablet_manager.cc:1888] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.398224 28295 log.cc:1196] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/e1141ea148d74ed594effc258ad44dd1
I20240501 19:56:49.398530 28295 ts_tablet_manager.cc:1909] T e1141ea148d74ed594effc258ad44dd1 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:49.399073 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d90e332364430d980a08be1f8a5c3a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:49.399202 24987 raft_consensus.cc:2454] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:49.399462 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet e1141ea148d74ed594effc258ad44dd1 (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.399827 24661 leader_election.cc:302] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:49.399977 26677 raft_consensus.cc:2781] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:49.400028 26677 raft_consensus.cc:482] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:49.400063 26677 raft_consensus.cc:3037] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.400151 28466 ts_tablet_manager.cc:1888] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.400208 28466 log.cc:1196] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/e1141ea148d74ed594effc258ad44dd1
I20240501 19:56:49.400442 28466 ts_tablet_manager.cc:1909] T e1141ea148d74ed594effc258ad44dd1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:49.401015 26677 raft_consensus.cc:504] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.401332 26677 leader_election.cc:288] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:49.401775 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d90e332364430d980a08be1f8a5c3a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:49.401803 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d90e332364430d980a08be1f8a5c3a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:49.401880 24987 raft_consensus.cc:3037] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.401424 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet a40a737a78824728b9e85e6a4ded638a (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.402024 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet e1141ea148d74ed594effc258ad44dd1 (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
W20240501 19:56:49.402206 24660 leader_election.cc:341] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:49.402573 27625 tablet_replica.cc:330] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:49.402702 27625 raft_consensus.cc:2229] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.402727 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet d56fe9e75a9c418f817185542ae4c1d9 (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.402801 24987 raft_consensus.cc:2454] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:49.403259 24661 leader_election.cc:302] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:49.403472 26677 raft_consensus.cc:2781] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:49.403566 26677 raft_consensus.cc:686] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:49.403707 26677 consensus_queue.cc:237] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.405138 29089 tablet_replica.cc:330] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:49.405282 29089 raft_consensus.cc:2229] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:49.405462 29089 raft_consensus.cc:2258] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.405889 29089 ts_tablet_manager.cc:1875] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.407626 29087 tablet_replica.cc:330] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:49.407893 29089 ts_tablet_manager.cc:1888] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.407883 29087 raft_consensus.cc:2229] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.407999 29087 raft_consensus.cc:2258] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.408016 29089 log.cc:1196] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/d56fe9e75a9c418f817185542ae4c1d9
I20240501 19:56:49.408300 29089 ts_tablet_manager.cc:1909] T d56fe9e75a9c418f817185542ae4c1d9 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:49.408398 24445 catalog_manager.cc:5484] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:49.408548 29054 tablet_bootstrap.cc:492] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:49.408690 29054 ts_tablet_manager.cc:1367] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20240501 19:56:49.408895 29089 tablet_replica.cc:330] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:49.409001 29089 raft_consensus.cc:2229] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:49.409139 29054 raft_consensus.cc:348] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.409240 29054 raft_consensus.cc:374] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.409272 29054 raft_consensus.cc:729] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.409154 29089 raft_consensus.cc:2258] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.409416 29054 consensus_queue.cc:260] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.409710 29054 ts_tablet_manager.cc:1398] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:49.409941 29089 ts_tablet_manager.cc:1875] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.411063 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet d56fe9e75a9c418f817185542ae4c1d9 (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.411020 29087 ts_tablet_manager.cc:1875] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.412070 29089 ts_tablet_manager.cc:1888] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.412176 29089 log.cc:1196] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/a40a737a78824728b9e85e6a4ded638a
I20240501 19:56:49.412460 29089 ts_tablet_manager.cc:1909] T a40a737a78824728b9e85e6a4ded638a P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:49.413146 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet a40a737a78824728b9e85e6a4ded638a (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.413599 29087 ts_tablet_manager.cc:1888] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.413715 29087 log.cc:1196] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/d56fe9e75a9c418f817185542ae4c1d9
I20240501 19:56:49.413992 29087 ts_tablet_manager.cc:1909] T d56fe9e75a9c418f817185542ae4c1d9 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:49.414822 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet d56fe9e75a9c418f817185542ae4c1d9 (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.402822 27625 raft_consensus.cc:2258] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.415468 27625 ts_tablet_manager.cc:1875] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.418859 27625 ts_tablet_manager.cc:1888] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.418989 27625 log.cc:1196] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/e1141ea148d74ed594effc258ad44dd1
I20240501 19:56:49.419303 27625 ts_tablet_manager.cc:1909] T e1141ea148d74ed594effc258ad44dd1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:49.423349 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet e1141ea148d74ed594effc258ad44dd1 (table rebalance_test_table_16 [id=c07fdbaa55724ad7aea8c41fd924a7e4]) successfully deleted
I20240501 19:56:49.441052 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:49.446182 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 6243d5812b16483ca3f2286d05920895 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a]), partition=
I20240501 19:56:49.446300 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 6243d5812b16483ca3f2286d05920895 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a]), partition=
I20240501 19:56:49.446391 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 6243d5812b16483ca3f2286d05920895 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a]), partition=
I20240501 19:56:49.446552 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6243d5812b16483ca3f2286d05920895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.446596 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6243d5812b16483ca3f2286d05920895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.446666 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6243d5812b16483ca3f2286d05920895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:49.448486 29059 tablet_bootstrap.cc:492] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:49.448660 29054 tablet_bootstrap.cc:492] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:49.449545 29059 tablet_bootstrap.cc:654] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.449558 29054 tablet_bootstrap.cc:654] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.449769 29104 tablet_bootstrap.cc:492] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:49.450610 29104 tablet_bootstrap.cc:654] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:49.451784 29104 tablet_bootstrap.cc:492] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:49.451877 29104 ts_tablet_manager.cc:1367] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:49.451994 29059 tablet_bootstrap.cc:492] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:49.452077 29059 ts_tablet_manager.cc:1367] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:49.452232 29104 raft_consensus.cc:348] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.452311 29104 raft_consensus.cc:374] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.452332 29104 raft_consensus.cc:729] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.452441 29104 consensus_queue.cc:260] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.452457 29059 raft_consensus.cc:348] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.452540 29059 raft_consensus.cc:374] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.452576 29059 raft_consensus.cc:729] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.452713 29104 ts_tablet_manager.cc:1398] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:49.452692 29059 consensus_queue.cc:260] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.452956 29059 ts_tablet_manager.cc:1398] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:49.453558 28856 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:49.453634 28856 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.453882 29054 tablet_bootstrap.cc:492] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:49.453967 29054 ts_tablet_manager.cc:1367] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:49.454010 28856 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:49.454308 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b747487a083433d91191aa79458f51e" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:49.454463 24987 raft_consensus.cc:2454] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:49.454751 27097 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:49.454732 29054 raft_consensus.cc:348] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.454840 29054 raft_consensus.cc:374] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:49.454895 29054 raft_consensus.cc:729] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.454941 28856 raft_consensus.cc:2781] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:49.455009 28856 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:49.455091 28856 raft_consensus.cc:3037] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.455015 29054 consensus_queue.cc:260] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.455075 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b747487a083433d91191aa79458f51e" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:49.455184 27425 raft_consensus.cc:2454] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:49.456142 28856 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.456467 28856 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:49.455376 29054 ts_tablet_manager.cc:1398] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:49.457789 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b747487a083433d91191aa79458f51e" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:49.457904 27425 raft_consensus.cc:3037] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.458778 27425 raft_consensus.cc:2454] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:49.458961 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b747487a083433d91191aa79458f51e" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:49.459110 24987 raft_consensus.cc:3037] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.459285 27098 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:49.459908 28856 raft_consensus.cc:2781] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:49.459992 28856 raft_consensus.cc:686] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:49.460121 28856 consensus_queue.cc:237] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.460156 24987 raft_consensus.cc:2454] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:49.460813 26885 raft_consensus.cc:482] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:49.460906 26885 raft_consensus.cc:504] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.461210 26885 leader_election.cc:288] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:49.461467 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6243d5812b16483ca3f2286d05920895" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:49.461578 27162 raft_consensus.cc:2454] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:49.461601 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6243d5812b16483ca3f2286d05920895" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:49.461704 27031 raft_consensus.cc:2454] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:49.461880 24531 leader_election.cc:302] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:49.462126 26885 raft_consensus.cc:2781] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:49.462203 26885 raft_consensus.cc:482] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:49.462239 26885 raft_consensus.cc:3037] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.462638 24451 catalog_manager.cc:5484] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:49.463178 26885 raft_consensus.cc:504] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.463507 26885 leader_election.cc:288] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:49.463985 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6243d5812b16483ca3f2286d05920895" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:49.464078 27162 raft_consensus.cc:3037] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.464164 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6243d5812b16483ca3f2286d05920895" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:49.464259 27031 raft_consensus.cc:3037] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.464987 27162 raft_consensus.cc:2454] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:49.465229 27031 raft_consensus.cc:2454] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:49.465517 24529 leader_election.cc:302] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:49.465682 26885 raft_consensus.cc:2781] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:49.465781 26885 raft_consensus.cc:686] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:49.465922 26885 consensus_queue.cc:237] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:49.467746 24451 catalog_manager.cc:5484] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:49.571740 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:49.576102 27031 raft_consensus.cc:1261] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:49.576203 27162 raft_consensus.cc:1261] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:49.576699 27943 consensus_queue.cc:1035] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.576968 27943 consensus_queue.cc:1035] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.579409 29114 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:49.579772 29114 tablet.cc:1620] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:49.581113 24451 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 6243d5812b16483ca3f2286d05920895 (table rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a]). Expected version 1 got 0
I20240501 19:56:49.582716 24451 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 6243d5812b16483ca3f2286d05920895 (table rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a]). Expected version 1 got 0
I20240501 19:56:49.583328 28923 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:49.583592 28923 tablet.cc:1620] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:49.585973 24446 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 6243d5812b16483ca3f2286d05920895 (table rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a]). Expected version 1 got 0
I20240501 19:56:49.587316 24451 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a] alter complete (version 1)
I20240501 19:56:49.589929 24454 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 6243d5812b16483ca3f2286d05920895 (table rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a]). Expected version 1 got 0
I20240501 19:56:49.589929 29116 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:49.590199 29116 tablet.cc:1620] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:49.652427 28518 raft_consensus.cc:482] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:49.652565 28518 raft_consensus.cc:504] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.652928 28542 raft_consensus.cc:482] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:49.653007 28518 leader_election.cc:288] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:49.653038 28542 raft_consensus.cc:504] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:49.653473 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:49.653555 28542 leader_election.cc:288] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:49.653509 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:49.653615 27276 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:49.653647 27425 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:49.653939 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d90e332364430d980a08be1f8a5c3a" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:49.654032 27098 leader_election.cc:302] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:49.654090 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d90e332364430d980a08be1f8a5c3a" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:49.654146 24987 raft_consensus.cc:2379] T 36d90e332364430d980a08be1f8a5c3a P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in current term 1: Already voted for candidate 9081292597be4a72a42b216fddcd7971 in this term.
I20240501 19:56:49.654181 28518 raft_consensus.cc:2781] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:49.654227 28518 raft_consensus.cc:482] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:49.654259 28518 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.654484 26965 leader_election.cc:302] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:56:49.654670 28542 raft_consensus.cc:3037] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.655406 28518 raft_consensus.cc:504] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.655548 28542 raft_consensus.cc:2726] T 36d90e332364430d980a08be1f8a5c3a P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:49.655736 28518 leader_election.cc:288] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:49.656086 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:49.656173 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:49.656219 27276 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.656283 27425 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:49.657168 27276 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:49.657168 27425 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:49.657440 27098 leader_election.cc:302] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:49.657620 28518 raft_consensus.cc:2781] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:49.657711 28518 raft_consensus.cc:686] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:49.657841 28518 consensus_queue.cc:237] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:49.659511 24451 catalog_manager.cc:5484] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:49.690183 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:49.691776 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:49.692107 27142 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:49.692889 24836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.693527 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:49.700778 29114 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:49.701117 29114 tablet.cc:1620] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:49.706794 29116 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:49.707129 29116 tablet.cc:1620] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:49.709022 28923 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:49.709335 28923 tablet.cc:1620] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:49.713001 24705 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.714313 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=91d686bb43f84d2394886edc5249563a] alter complete (version 2)
I20240501 19:56:49.722606 27012 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:49.728154 24575 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:49.728181 24968 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:49.825055 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:49.833278 29114 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:49.835233 28923 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:49.836097 29116 tablet.cc:1722] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:49.839596 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=91d686bb43f84d2394886edc5249563a] alter complete (version 3)
I20240501 19:56:49.846123 27142 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:49.856056 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:49.856230 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:49.857326 24836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.857332 24705 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.877810 24968 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:49.879544 24575 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:49.880563 27012 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:49.887261 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 19:56:49.887401 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 19:56:49.891227 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 4 replicas of tablet 0d91e1c792c749e7883bb996d58d99ea
I20240501 19:56:49.891790 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 6fe0314450814a31ab1bdda90ab245e2
I20240501 19:56:49.892002 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 7beb152e6d5a435eb9c97fa1cace4b39
I20240501 19:56:49.892530 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d91e1c792c749e7883bb996d58d99ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.892854 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d91e1c792c749e7883bb996d58d99ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:49.893326 28295 tablet_replica.cc:330] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.893460 28295 raft_consensus.cc:2229] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:49.893674 28295 raft_consensus.cc:2258] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.894166 28295 ts_tablet_manager.cc:1875] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.895182 29089 tablet_replica.cc:330] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:49.895318 29089 raft_consensus.cc:2229] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.895450 29089 raft_consensus.cc:2258] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.895650 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d91e1c792c749e7883bb996d58d99ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:49.895762 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe0314450814a31ab1bdda90ab245e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:49.895815 27625 tablet_replica.cc:330] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:49.895885 24967 tablet_service.cc:1514] Processing DeleteTablet for tablet 7beb152e6d5a435eb9c97fa1cace4b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:49.895891 29089 ts_tablet_manager.cc:1875] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.895937 27625 raft_consensus.cc:2229] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.896073 27625 raft_consensus.cc:2258] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.896492 28295 ts_tablet_manager.cc:1888] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.896514 27625 ts_tablet_manager.cc:1875] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.896554 28295 log.cc:1196] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d91e1c792c749e7883bb996d58d99ea
I20240501 19:56:49.896814 28295 ts_tablet_manager.cc:1909] T 0d91e1c792c749e7883bb996d58d99ea P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:49.898101 29089 ts_tablet_manager.cc:1888] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.898160 29089 log.cc:1196] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d91e1c792c749e7883bb996d58d99ea
I20240501 19:56:49.898422 29089 ts_tablet_manager.cc:1909] T 0d91e1c792c749e7883bb996d58d99ea P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:49.898739 27625 ts_tablet_manager.cc:1888] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.898799 27625 log.cc:1196] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/0d91e1c792c749e7883bb996d58d99ea
I20240501 19:56:49.899068 27625 ts_tablet_manager.cc:1909] T 0d91e1c792c749e7883bb996d58d99ea P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:49.899073 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d91e1c792c749e7883bb996d58d99ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:49.899400 29180 tablet_replica.cc:330] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:49.899673 29180 raft_consensus.cc:2229] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.899749 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 0d91e1c792c749e7883bb996d58d99ea (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.899780 29180 raft_consensus.cc:2258] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.899947 27625 tablet_replica.cc:330] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:49.900043 27625 raft_consensus.cc:2229] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:49.900197 27625 raft_consensus.cc:2258] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.900257 29180 ts_tablet_manager.cc:1875] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.900709 27625 ts_tablet_manager.cc:1875] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.901459 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 0d91e1c792c749e7883bb996d58d99ea (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.901572 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 7beb152e6d5a435eb9c97fa1cace4b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:49.901592 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 0d91e1c792c749e7883bb996d58d99ea (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.901729 29089 tablet_replica.cc:330] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:49.901827 29089 raft_consensus.cc:2229] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.901880 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe0314450814a31ab1bdda90ab245e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.901928 29089 raft_consensus.cc:2258] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.901955 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 7beb152e6d5a435eb9c97fa1cace4b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.902020 28295 tablet_replica.cc:330] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.902143 28295 raft_consensus.cc:2229] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.902253 28295 raft_consensus.cc:2258] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.902371 29089 ts_tablet_manager.cc:1875] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.902730 28295 ts_tablet_manager.cc:1875] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.902931 27625 ts_tablet_manager.cc:1888] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.902990 27625 log.cc:1196] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/6fe0314450814a31ab1bdda90ab245e2
I20240501 19:56:49.903295 27625 ts_tablet_manager.cc:1909] T 6fe0314450814a31ab1bdda90ab245e2 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:49.904151 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 6fe0314450814a31ab1bdda90ab245e2 (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.904330 27625 tablet_replica.cc:330] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:49.904428 27625 raft_consensus.cc:2229] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:49.904568 27625 raft_consensus.cc:2258] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.904631 28295 ts_tablet_manager.cc:1888] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.904625 29089 ts_tablet_manager.cc:1888] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.904687 28295 log.cc:1196] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/6fe0314450814a31ab1bdda90ab245e2
I20240501 19:56:49.904744 29089 log.cc:1196] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/7beb152e6d5a435eb9c97fa1cace4b39
I20240501 19:56:49.904948 28295 ts_tablet_manager.cc:1909] T 6fe0314450814a31ab1bdda90ab245e2 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:49.904955 27625 ts_tablet_manager.cc:1875] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.904994 29089 ts_tablet_manager.cc:1909] T 7beb152e6d5a435eb9c97fa1cace4b39 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:49.905344 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe0314450814a31ab1bdda90ab245e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:49.905526 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 7beb152e6d5a435eb9c97fa1cace4b39 (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.906606 27625 ts_tablet_manager.cc:1888] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.906668 27625 log.cc:1196] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/7beb152e6d5a435eb9c97fa1cace4b39
I20240501 19:56:49.906893 27625 ts_tablet_manager.cc:1909] T 7beb152e6d5a435eb9c97fa1cace4b39 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:49.907238 28295 tablet_replica.cc:330] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.907275 29180 ts_tablet_manager.cc:1888] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.907331 29180 log.cc:1196] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/0d91e1c792c749e7883bb996d58d99ea
I20240501 19:56:49.907367 28295 raft_consensus.cc:2229] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.907486 28295 raft_consensus.cc:2258] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.907601 29180 ts_tablet_manager.cc:1909] T 0d91e1c792c749e7883bb996d58d99ea P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:49.908211 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 0d91e1c792c749e7883bb996d58d99ea (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.908279 29181 tablet_replica.cc:330] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:49.908301 28295 ts_tablet_manager.cc:1875] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.908211 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 6fe0314450814a31ab1bdda90ab245e2 (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.908406 29181 raft_consensus.cc:2229] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.908527 29181 raft_consensus.cc:2258] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.908677 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 7beb152e6d5a435eb9c97fa1cace4b39 (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.909201 29181 ts_tablet_manager.cc:1875] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.910157 28295 ts_tablet_manager.cc:1888] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.910230 28295 log.cc:1196] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/7beb152e6d5a435eb9c97fa1cace4b39
I20240501 19:56:49.910496 28295 ts_tablet_manager.cc:1909] T 7beb152e6d5a435eb9c97fa1cace4b39 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:49.911326 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 7beb152e6d5a435eb9c97fa1cace4b39 (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.911829 29181 ts_tablet_manager.cc:1888] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:49.911932 29181 log.cc:1196] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/6fe0314450814a31ab1bdda90ab245e2
I20240501 19:56:49.912230 29181 ts_tablet_manager.cc:1909] T 6fe0314450814a31ab1bdda90ab245e2 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:49.918767 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 6fe0314450814a31ab1bdda90ab245e2 (table rebalance_test_table_18 [id=4d222a7ce3414babb32fd3a6232092c0]) successfully deleted
I20240501 19:56:49.944293 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:49.944425 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:49.946702 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 6243d5812b16483ca3f2286d05920895
I20240501 19:56:49.947289 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 6243d5812b16483ca3f2286d05920895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:49.947602 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet 6243d5812b16483ca3f2286d05920895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:49.947620 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 6243d5812b16483ca3f2286d05920895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:49 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.947839 28295 tablet_replica.cc:330] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.947773 24857 consensus_queue.cc:237] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:49.947894 29180 tablet_replica.cc:330] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:49.947911 29197 tablet_replica.cc:330] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:49.947979 28295 raft_consensus.cc:2229] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:49.948005 29180 raft_consensus.cc:2229] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.948029 29197 raft_consensus.cc:2229] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.948109 29180 raft_consensus.cc:2258] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.948136 29197 raft_consensus.cc:2258] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.948185 28295 raft_consensus.cc:2258] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.949028 28295 ts_tablet_manager.cc:1875] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.949174 24987 raft_consensus.cc:1261] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:49.949198 27276 raft_consensus.cc:1261] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:49.949234 29197 ts_tablet_manager.cc:1875] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.949586 29168 consensus_queue.cc:1035] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:49.950080 29168 consensus_queue.cc:1035] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:49.951328 28295 ts_tablet_manager.cc:1888] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.951406 28295 log.cc:1196] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/6243d5812b16483ca3f2286d05920895
I20240501 19:56:49.951545 29180 ts_tablet_manager.cc:1875] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:49.951684 28295 ts_tablet_manager.cc:1909] T 6243d5812b16483ca3f2286d05920895 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:49.952286 29197 ts_tablet_manager.cc:1888] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.952350 29197 log.cc:1196] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/6243d5812b16483ca3f2286d05920895
I20240501 19:56:49.952616 29197 ts_tablet_manager.cc:1909] T 6243d5812b16483ca3f2286d05920895 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:49.953591 29180 ts_tablet_manager.cc:1888] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:49.953646 29180 log.cc:1196] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/6243d5812b16483ca3f2286d05920895
I20240501 19:56:49.953881 29180 ts_tablet_manager.cc:1909] T 6243d5812b16483ca3f2286d05920895 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:49.954704 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 6243d5812b16483ca3f2286d05920895 (table rebalancer_dynamic_table_new_name [id=91d686bb43f84d2394886edc5249563a]) successfully deleted
I20240501 19:56:49.955978 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 6243d5812b16483ca3f2286d05920895 (table rebalancer_dynamic_table_new_name [id=91d686bb43f84d2394886edc5249563a]) successfully deleted
I20240501 19:56:49.956609 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 6243d5812b16483ca3f2286d05920895 (table rebalancer_dynamic_table_new_name [id=91d686bb43f84d2394886edc5249563a]) successfully deleted
I20240501 19:56:49.957186 28114 raft_consensus.cc:2932] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.957675 27276 raft_consensus.cc:2932] T 0f8f81836b9944aa964a8236912c0183 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.959507 24987 raft_consensus.cc:2932] T 0f8f81836b9944aa964a8236912c0183 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.959980 24454 catalog_manager.cc:5484] T 0f8f81836b9944aa964a8236912c0183 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:49.963069 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8f81836b9944aa964a8236912c0183 with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:49.963670 28295 tablet_replica.cc:330] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:49.963815 28295 raft_consensus.cc:2229] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:49.963997 28295 raft_consensus.cc:2258] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:49.964522 28295 ts_tablet_manager.cc:1875] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:49.966364 28709 consensus_queue.cc:1035] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.966536 28295 ts_tablet_manager.cc:1888] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:49.966610 28295 log.cc:1196] T 0f8f81836b9944aa964a8236912c0183 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f8f81836b9944aa964a8236912c0183
I20240501 19:56:49.967113 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 0f8f81836b9944aa964a8236912c0183 (table rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]) successfully deleted
I20240501 19:56:49.969656 28709 consensus_queue.cc:1035] T 36d90e332364430d980a08be1f8a5c3a P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.974911 28518 consensus_queue.cc:1035] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:49.977717 28518 consensus_queue.cc:1035] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.040201 27142 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:50.041292 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:50.046625 24705 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.052788 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:50.055527 27012 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:50.055509 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:50.068075 24967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.068142 24574 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.069295 27406 tablet_service.cc:1467] Processing CreateTablet for tablet b6a84bec086a4dbab9e616a0ace7f309 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f1652654ef94ec89952c206d8d00e2d]), partition=
I20240501 19:56:50.069599 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a84bec086a4dbab9e616a0ace7f309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.069804 24836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.070675 24574 tablet_service.cc:1467] Processing CreateTablet for tablet b6a84bec086a4dbab9e616a0ace7f309 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f1652654ef94ec89952c206d8d00e2d]), partition=
I20240501 19:56:50.070914 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a84bec086a4dbab9e616a0ace7f309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.070931 27142 tablet_service.cc:1467] Processing CreateTablet for tablet b6a84bec086a4dbab9e616a0ace7f309 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f1652654ef94ec89952c206d8d00e2d]), partition=
I20240501 19:56:50.071192 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a84bec086a4dbab9e616a0ace7f309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.074113 29258 tablet_bootstrap.cc:492] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:50.075137 29258 tablet_bootstrap.cc:654] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.079337 29259 tablet_bootstrap.cc:492] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:50.088112 29257 tablet_bootstrap.cc:492] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:50.088461 29259 tablet_bootstrap.cc:654] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.089982 29257 tablet_bootstrap.cc:654] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.090759 29259 tablet_bootstrap.cc:492] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:50.091500 29259 ts_tablet_manager.cc:1367] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:50.092031 29259 raft_consensus.cc:348] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.092135 29259 raft_consensus.cc:374] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.092176 29259 raft_consensus.cc:729] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.092350 29259 consensus_queue.cc:260] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.092767 29257 tablet_bootstrap.cc:492] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:50.092855 29257 ts_tablet_manager.cc:1367] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:50.093217 29257 raft_consensus.cc:348] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.093305 29257 raft_consensus.cc:374] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.093349 29257 raft_consensus.cc:729] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.093472 29257 consensus_queue.cc:260] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.093717 29257 ts_tablet_manager.cc:1398] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:50.094014 29259 ts_tablet_manager.cc:1398] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:50.098867 29258 tablet_bootstrap.cc:492] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:50.098989 29258 ts_tablet_manager.cc:1367] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.002s
I20240501 19:56:50.099414 29258 raft_consensus.cc:348] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.099503 29258 raft_consensus.cc:374] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.099529 29258 raft_consensus.cc:729] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.099761 29258 consensus_queue.cc:260] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.100019 29258 ts_tablet_manager.cc:1398] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:50.131259 24594 consensus_queue.cc:237] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.132490 27425 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.132531 24987 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.132836 29115 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.133028 26885 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.135046 29115 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.135383 24987 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.137784 24445 catalog_manager.cc:5484] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.137845 27425 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.141418 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 718c7f2f1f074e3c84534b502a4809a2 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:50.141707 29181 tablet_replica.cc:330] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:50.141871 29181 raft_consensus.cc:2229] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:50.142045 29181 raft_consensus.cc:2258] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:50.142563 29181 ts_tablet_manager.cc:1875] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:50.144544 29181 ts_tablet_manager.cc:1888] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:50.144594 29181 log.cc:1196] T 718c7f2f1f074e3c84534b502a4809a2 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/718c7f2f1f074e3c84534b502a4809a2
I20240501 19:56:50.145037 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 718c7f2f1f074e3c84534b502a4809a2 (table rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]) successfully deleted
I20240501 19:56:50.206727 26885 raft_consensus.cc:482] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:50.206847 26885 raft_consensus.cc:504] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.207304 26885 leader_election.cc:288] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:50.207895 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a84bec086a4dbab9e616a0ace7f309" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:50.208042 27162 raft_consensus.cc:2454] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:50.208029 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a84bec086a4dbab9e616a0ace7f309" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:50.208166 27425 raft_consensus.cc:2454] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:50.208499 24531 leader_election.cc:302] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:50.208668 26885 raft_consensus.cc:2781] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:50.208732 26885 raft_consensus.cc:482] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:50.208774 26885 raft_consensus.cc:3037] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.209913 26885 raft_consensus.cc:504] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.210237 26885 leader_election.cc:288] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:50.210559 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a84bec086a4dbab9e616a0ace7f309" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:50.210618 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a84bec086a4dbab9e616a0ace7f309" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:50.210673 27162 raft_consensus.cc:3037] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.210731 27425 raft_consensus.cc:3037] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.211666 27162 raft_consensus.cc:2454] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:50.211748 27425 raft_consensus.cc:2454] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:50.212014 24531 leader_election.cc:302] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:50.212193 26885 raft_consensus.cc:2781] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:50.212298 26885 raft_consensus.cc:686] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:50.212451 26885 consensus_queue.cc:237] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.214170 24452 catalog_manager.cc:5484] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.225899 28518 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.231076 28856 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.233862 24452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:50.234369 27012 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:50.236200 27142 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:50.237779 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:50.238050 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:50.244091 27406 tablet_service.cc:1467] Processing CreateTablet for tablet d90582276fb3409b984a4bd96f25e291 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.244158 27274 tablet_service.cc:1467] Processing CreateTablet for tablet d90582276fb3409b984a4bd96f25e291 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.244403 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90582276fb3409b984a4bd96f25e291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.244454 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90582276fb3409b984a4bd96f25e291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.246690 29257 tablet_bootstrap.cc:492] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:50.246858 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 2a44d1713369478191bfc30457bf61c1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.247128 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a44d1713369478191bfc30457bf61c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.247227 27012 tablet_service.cc:1467] Processing CreateTablet for tablet d29130c48f7a435498e854973a090b37 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.247501 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29130c48f7a435498e854973a090b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.247694 29257 tablet_bootstrap.cc:654] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.248867 24705 tablet_service.cc:1467] Processing CreateTablet for tablet d90582276fb3409b984a4bd96f25e291 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.249173 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90582276fb3409b984a4bd96f25e291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.249547 24836 tablet_service.cc:1467] Processing CreateTablet for tablet d29130c48f7a435498e854973a090b37 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.249802 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29130c48f7a435498e854973a090b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.252306 29338 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:50.252295 24574 tablet_service.cc:1467] Processing CreateTablet for tablet d29130c48f7a435498e854973a090b37 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.252591 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29130c48f7a435498e854973a090b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.252970 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 2a44d1713369478191bfc30457bf61c1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.253194 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a44d1713369478191bfc30457bf61c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.253733 29338 tablet_bootstrap.cc:654] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.255296 29335 tablet_bootstrap.cc:492] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:50.256304 29258 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:50.257092 29258 tablet_bootstrap.cc:654] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.258072 24705 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.258078 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 2a44d1713369478191bfc30457bf61c1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.258338 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a44d1713369478191bfc30457bf61c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.258872 29257 tablet_bootstrap.cc:492] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:50.258956 29257 ts_tablet_manager.cc:1367] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:50.259469 29257 raft_consensus.cc:348] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.259578 29257 raft_consensus.cc:374] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.259611 29257 raft_consensus.cc:729] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.259732 29257 consensus_queue.cc:260] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.259987 29257 ts_tablet_manager.cc:1398] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.261559 29339 tablet_bootstrap.cc:492] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:50.261621 29259 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:50.261865 29335 tablet_bootstrap.cc:654] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.262619 29259 tablet_bootstrap.cc:654] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.268795 29258 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:50.268889 29258 ts_tablet_manager.cc:1367] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240501 19:56:50.269311 29258 raft_consensus.cc:348] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.269408 29258 raft_consensus.cc:374] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.269435 29258 raft_consensus.cc:729] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.269537 29258 consensus_queue.cc:260] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.269788 29258 ts_tablet_manager.cc:1398] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:50.271831 24574 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:50.273753 29259 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:50.273854 29259 ts_tablet_manager.cc:1367] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:50.274312 29259 raft_consensus.cc:348] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.274412 29259 raft_consensus.cc:374] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.274441 29259 raft_consensus.cc:729] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.274545 29259 consensus_queue.cc:260] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.274818 29259 ts_tablet_manager.cc:1398] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.276706 24967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.279624 24836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.284595 29339 tablet_bootstrap.cc:654] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.285732 29335 tablet_bootstrap.cc:492] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:50.285878 29335 ts_tablet_manager.cc:1367] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.001s
I20240501 19:56:50.286348 29335 raft_consensus.cc:348] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.286563 29335 raft_consensus.cc:374] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.286630 29335 raft_consensus.cc:729] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.286823 29335 consensus_queue.cc:260] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.287210 29335 ts_tablet_manager.cc:1398] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.287731 29335 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:50.288604 29335 tablet_bootstrap.cc:654] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.289678 29337 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:50.290745 29337 tablet_bootstrap.cc:654] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.292145 29335 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:50.292289 29335 ts_tablet_manager.cc:1367] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 19:56:50.292835 29335 raft_consensus.cc:348] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.292989 29335 raft_consensus.cc:374] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.293058 29335 raft_consensus.cc:729] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.293236 29335 consensus_queue.cc:260] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.293944 29335 ts_tablet_manager.cc:1398] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 19:56:50.294611 29337 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:50.294746 29337 ts_tablet_manager.cc:1367] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:50.294962 29338 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:50.295218 29337 raft_consensus.cc:348] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.295387 29337 raft_consensus.cc:374] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.295459 29337 raft_consensus.cc:729] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.295650 29337 consensus_queue.cc:260] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.296466 29337 ts_tablet_manager.cc:1398] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:50.296870 29338 ts_tablet_manager.cc:1367] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20240501 19:56:50.296762 29337 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:50.297310 29338 raft_consensus.cc:348] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.297844 29337 tablet_bootstrap.cc:654] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.297807 29338 raft_consensus.cc:374] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.298556 29338 raft_consensus.cc:729] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.299597 29337 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:50.299701 29337 ts_tablet_manager.cc:1367] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:50.299721 29338 consensus_queue.cc:260] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.300145 29337 raft_consensus.cc:348] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.300248 29337 raft_consensus.cc:374] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.300283 29337 raft_consensus.cc:729] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.300387 29337 consensus_queue.cc:260] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.300537 29339 tablet_bootstrap.cc:492] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:50.300621 29339 ts_tablet_manager.cc:1367] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240501 19:56:50.300652 29337 ts_tablet_manager.cc:1398] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.301759 29339 raft_consensus.cc:348] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.302937 29339 raft_consensus.cc:374] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.303299 29339 raft_consensus.cc:729] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.307363 29339 consensus_queue.cc:260] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.307693 29339 ts_tablet_manager.cc:1398] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:50.308842 29338 ts_tablet_manager.cc:1398] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240501 19:56:50.336160 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:50.338492 28709 raft_consensus.cc:482] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:50.338608 28709 raft_consensus.cc:504] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.339018 28709 leader_election.cc:288] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:50.339382 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90582276fb3409b984a4bd96f25e291" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:50.339530 27276 raft_consensus.cc:2454] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:50.339911 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90582276fb3409b984a4bd96f25e291" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:50.339987 24662 leader_election.cc:302] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:50.340045 27424 raft_consensus.cc:2454] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:50.340167 28709 raft_consensus.cc:2781] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:50.340243 28709 raft_consensus.cc:482] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:50.340277 28709 raft_consensus.cc:3037] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.341338 28709 raft_consensus.cc:504] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.341679 28709 leader_election.cc:288] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:50.341861 27162 raft_consensus.cc:1261] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:50.341986 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90582276fb3409b984a4bd96f25e291" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:50.342125 27425 raft_consensus.cc:3037] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.342183 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90582276fb3409b984a4bd96f25e291" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:50.342305 27276 raft_consensus.cc:3037] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.342434 27424 raft_consensus.cc:1261] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:50.343127 26885 consensus_queue.cc:1035] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.343269 27425 raft_consensus.cc:2454] T d90582276fb3409b984a4bd96f25e291 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:50.343533 24662 leader_election.cc:302] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:50.343572 27276 raft_consensus.cc:2454] T d90582276fb3409b984a4bd96f25e291 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:50.343750 26885 consensus_queue.cc:1035] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.344247 28709 raft_consensus.cc:2781] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:50.344327 28709 raft_consensus.cc:686] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:50.344447 28709 consensus_queue.cc:237] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.346448 24450 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet b6a84bec086a4dbab9e616a0ace7f309 (table rebalancer_dynamic_table [id=9f1652654ef94ec89952c206d8d00e2d]). Expected version 1 got 0
I20240501 19:56:50.346858 24451 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet b6a84bec086a4dbab9e616a0ace7f309 (table rebalancer_dynamic_table [id=9f1652654ef94ec89952c206d8d00e2d]). Expected version 1 got 0
I20240501 19:56:50.347101 29368 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:50.346529 29367 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:50.347440 29368 tablet.cc:1620] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:50.347690 29367 tablet.cc:1620] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:50.349434 24452 catalog_manager.cc:5484] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:50.351398 24594 consensus_queue.cc:237] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.351994 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=9f1652654ef94ec89952c206d8d00e2d] alter complete (version 1)
I20240501 19:56:50.352977 24987 raft_consensus.cc:1261] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.353266 27424 raft_consensus.cc:1261] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.353808 29115 consensus_queue.cc:1035] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.353966 27943 consensus_queue.cc:1035] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.355875 24452 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet b6a84bec086a4dbab9e616a0ace7f309 (table rebalancer_dynamic_table [id=9f1652654ef94ec89952c206d8d00e2d]). Expected version 1 got 0
I20240501 19:56:50.356577 29115 raft_consensus.cc:2932] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.356956 29371 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:50.357265 29371 tablet.cc:1620] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:50.357852 27425 raft_consensus.cc:2932] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.358665 24987 raft_consensus.cc:2932] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.360268 24452 catalog_manager.cc:5484] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.362895 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:50.363174 29181 tablet_replica.cc:330] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:50.363363 29181 raft_consensus.cc:2229] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:50.363535 29181 raft_consensus.cc:2258] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:50.363984 29181 ts_tablet_manager.cc:1875] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:50.365701 29181 ts_tablet_manager.cc:1888] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:50.365752 29181 log.cc:1196] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/71a4aa3f94bf4a3b8d84cae1e1765ac9
I20240501 19:56:50.366266 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9 (table rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]) successfully deleted
I20240501 19:56:50.369521 28518 raft_consensus.cc:482] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:50.369623 28518 raft_consensus.cc:504] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.370064 28518 leader_election.cc:288] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:50.371201 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44d1713369478191bfc30457bf61c1" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:50.371372 27276 raft_consensus.cc:2454] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:50.372171 27098 leader_election.cc:302] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:50.372339 28518 raft_consensus.cc:2781] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:50.372390 28518 raft_consensus.cc:482] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:50.372435 28518 raft_consensus.cc:3037] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.373488 28518 raft_consensus.cc:504] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.373859 28518 leader_election.cc:288] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:50.374428 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44d1713369478191bfc30457bf61c1" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:50.374462 28542 raft_consensus.cc:482] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:50.374542 27276 raft_consensus.cc:3037] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.374549 28542 raft_consensus.cc:504] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.374955 28542 leader_election.cc:288] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:50.375804 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:50.375926 24594 raft_consensus.cc:2454] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:50.376117 27276 raft_consensus.cc:2454] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:50.376271 26965 leader_election.cc:302] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:50.376499 28542 raft_consensus.cc:2781] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:50.376552 28542 raft_consensus.cc:482] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:50.376585 28542 raft_consensus.cc:3037] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.376559 27098 leader_election.cc:302] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:50.376935 28518 raft_consensus.cc:2781] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:50.377025 28518 raft_consensus.cc:686] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:50.377149 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44d1713369478191bfc30457bf61c1" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:50.377245 27032 raft_consensus.cc:3037] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.377195 28518 consensus_queue.cc:237] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.377471 28542 raft_consensus.cc:504] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.377800 28542 leader_election.cc:288] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:50.377965 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44d1713369478191bfc30457bf61c1" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:50.378224 27032 raft_consensus.cc:2454] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:50.378482 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:50.378597 24594 raft_consensus.cc:3037] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.379158 28114 raft_consensus.cc:482] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:50.379249 28114 raft_consensus.cc:504] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.379411 24451 catalog_manager.cc:5484] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:50.379573 24594 raft_consensus.cc:2454] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:50.379752 28114 leader_election.cc:288] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:50.379851 26965 leader_election.cc:302] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:50.380250 28542 raft_consensus.cc:2781] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:50.380402 28542 raft_consensus.cc:686] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:50.380540 28542 consensus_queue.cc:237] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.382165 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:50.382295 24594 raft_consensus.cc:2379] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d64cdfaeb2d46b29845160e395fc241 in current term 1: Already voted for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in this term.
I20240501 19:56:50.382227 24451 catalog_manager.cc:5484] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:50.383778 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:50.383899 24856 raft_consensus.cc:2454] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:50.384091 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:50.384187 24857 raft_consensus.cc:3037] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.385232 24857 raft_consensus.cc:2454] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:50.391667 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:50.392026 24791 leader_election.cc:302] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:56:50.392359 28114 raft_consensus.cc:2726] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:50.456579 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:50.462136 29367 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:50.462515 29367 tablet.cc:1620] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:50.462844 29371 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:50.463303 29371 tablet.cc:1620] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:50.464936 29368 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:50.465196 29368 tablet.cc:1620] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:50.468961 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=9f1652654ef94ec89952c206d8d00e2d] alter complete (version 2)
I20240501 19:56:50.477370 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:50.478572 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:50.484769 24836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.487625 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:50.490694 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:50.491925 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:50.493299 24574 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:50.512779 24967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.571367 27032 consensus_queue.cc:237] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:50.573338 24594 raft_consensus.cc:1261] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:50.573418 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240501 19:56:50.573671 26966 consensus_peers.cc:485] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51cd33a27a0a4754bbe044b0ff0ef262. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.573773 27162 raft_consensus.cc:1261] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:50.574177 29384 consensus_queue.cc:1035] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:50.575263 29442 consensus_queue.cc:1035] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:50.577982 29442 raft_consensus.cc:2932] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:50.578378 27162 raft_consensus.cc:2932] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:50.580760 24594 raft_consensus.cc:2932] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:50.581285 24454 catalog_manager.cc:5484] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:50.586813 29367 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:50.587383 29368 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:50.591841 29371 tablet.cc:1722] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:50.593238 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=9f1652654ef94ec89952c206d8d00e2d] alter complete (version 3)
I20240501 19:56:50.609050 27276 consensus_queue.cc:237] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:50.610385 27032 raft_consensus.cc:1261] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:50.610447 27162 raft_consensus.cc:1261] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:50.610567 27227 consensus_peers.cc:485] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4851f59ff041424bae97a3ecaa9e191b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.610796 28334 consensus_queue.cc:1035] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:50.610971 29461 consensus_queue.cc:1035] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:50.612697 29461 raft_consensus.cc:2932] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:50.613137 27032 raft_consensus.cc:2932] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:50.614797 27162 raft_consensus.cc:2932] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:50.615540 24454 catalog_manager.cc:5484] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:50.637742 24987 consensus_queue.cc:237] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:50.639602 24857 raft_consensus.cc:1261] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.639654 24594 raft_consensus.cc:1261] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:50.639734 24922 consensus_peers.cc:485] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23ae3af247c1456e8b75640c25906faf. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.640003 29210 consensus_queue.cc:1035] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.640190 29030 consensus_queue.cc:1035] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.642035 29210 raft_consensus.cc:2932] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:50.642436 24857 raft_consensus.cc:2932] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:50.644028 24594 raft_consensus.cc:2932] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:50.645282 24446 catalog_manager.cc:5484] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:50.666007 24727 consensus_queue.cc:237] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:50.667488 24857 raft_consensus.cc:1261] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.667533 24987 raft_consensus.cc:1261] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:50.668002 24662 consensus_peers.cc:485] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 164460e6fbc44fb1a9a12d52fea9254b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.668017 28709 consensus_queue.cc:1035] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.668434 29500 consensus_queue.cc:1035] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.670059 28709 raft_consensus.cc:2932] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:50.671407 24857 raft_consensus.cc:2932] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:50.673179 24454 catalog_manager.cc:5484] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:50.674317 24987 raft_consensus.cc:2932] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:50.690718 24857 consensus_queue.cc:237] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:50.692183 24987 raft_consensus.cc:1261] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:50.692291 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b692b5b96534e79a32a8b06b92f32d6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.692497 24594 raft_consensus.cc:1261] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.692765 29520 consensus_queue.cc:1035] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.693120 29519 consensus_queue.cc:1035] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.695070 29520 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:50.695487 24594 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:50.696694 24446 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:50.696825 24446 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:50.698943 24454 catalog_manager.cc:5484] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:50.700024 24446 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet b6a84bec086a4dbab9e616a0ace7f309
I20240501 19:56:50.700564 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a84bec086a4dbab9e616a0ace7f309 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:50 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:50.700840 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a84bec086a4dbab9e616a0ace7f309 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:50.701167 27406 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a84bec086a4dbab9e616a0ace7f309 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:50 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:50.701205 29525 tablet_replica.cc:330] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:50.701346 29525 raft_consensus.cc:2229] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:50.701643 29524 tablet_replica.cc:330] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:50.701856 29524 raft_consensus.cc:2229] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:50.701886 24987 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:50.702215 29524 raft_consensus.cc:2258] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:50.702734 29525 raft_consensus.cc:2258] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:50.703472 29524 ts_tablet_manager.cc:1875] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:50.703598 29525 ts_tablet_manager.cc:1875] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:50.705566 29524 ts_tablet_manager.cc:1888] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:50.705689 29524 log.cc:1196] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/b6a84bec086a4dbab9e616a0ace7f309
I20240501 19:56:50.706045 29524 ts_tablet_manager.cc:1909] T b6a84bec086a4dbab9e616a0ace7f309 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:50.706467 29525 ts_tablet_manager.cc:1888] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:50.706555 29525 log.cc:1196] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/b6a84bec086a4dbab9e616a0ace7f309
I20240501 19:56:50.706830 29525 ts_tablet_manager.cc:1909] T b6a84bec086a4dbab9e616a0ace7f309 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:50.706800 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet b6a84bec086a4dbab9e616a0ace7f309 (table rebalancer_dynamic_table_new_name [id=9f1652654ef94ec89952c206d8d00e2d]) successfully deleted
I20240501 19:56:50.707216 29526 tablet_replica.cc:330] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:50.707401 29526 raft_consensus.cc:2229] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:50.707567 29526 raft_consensus.cc:2258] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:50.707619 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet b6a84bec086a4dbab9e616a0ace7f309 (table rebalancer_dynamic_table_new_name [id=9f1652654ef94ec89952c206d8d00e2d]) successfully deleted
I20240501 19:56:50.708070 29526 ts_tablet_manager.cc:1875] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:50.710292 29526 ts_tablet_manager.cc:1888] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:50.710357 29526 log.cc:1196] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/b6a84bec086a4dbab9e616a0ace7f309
I20240501 19:56:50.710604 29526 ts_tablet_manager.cc:1909] T b6a84bec086a4dbab9e616a0ace7f309 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:50.712158 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet b6a84bec086a4dbab9e616a0ace7f309 (table rebalancer_dynamic_table_new_name [id=9f1652654ef94ec89952c206d8d00e2d]) successfully deleted
I20240501 19:56:50.722046 24857 consensus_queue.cc:237] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:50.723560 24727 raft_consensus.cc:1261] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.723626 24987 raft_consensus.cc:1261] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:50.723822 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92dd4bc514f146bb89083fe8a9315308. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.724009 29523 consensus_queue.cc:1035] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.724169 29519 consensus_queue.cc:1035] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.725800 29523 raft_consensus.cc:2932] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.726277 24987 raft_consensus.cc:2932] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.728358 24727 raft_consensus.cc:2932] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.729645 24448 catalog_manager.cc:5484] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:50.741804 29500 consensus_queue.cc:1035] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.752844 28709 consensus_queue.cc:1035] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.788435 29442 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.792717 29384 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.802026 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:50.807498 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f69515b0b8c547c0a4b5f0521f4a167a]), partition=
I20240501 19:56:50.807570 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f69515b0b8c547c0a4b5f0521f4a167a]), partition=
I20240501 19:56:50.807690 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f69515b0b8c547c0a4b5f0521f4a167a]), partition=
I20240501 19:56:50.807777 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.807826 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.807909 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.809619 29339 tablet_bootstrap.cc:492] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:50.809939 29592 tablet_bootstrap.cc:492] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:50.810403 29591 tablet_bootstrap.cc:492] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:50.810495 29339 tablet_bootstrap.cc:654] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.811180 29592 tablet_bootstrap.cc:654] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.812191 29339 tablet_bootstrap.cc:492] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:50.812274 29339 ts_tablet_manager.cc:1367] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:50.812489 29591 tablet_bootstrap.cc:654] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.812721 29339 raft_consensus.cc:348] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.812827 29339 raft_consensus.cc:374] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.812861 29339 raft_consensus.cc:729] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.812984 29339 consensus_queue.cc:260] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.813164 29592 tablet_bootstrap.cc:492] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:50.813251 29592 ts_tablet_manager.cc:1367] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:50.813274 29339 ts_tablet_manager.cc:1398] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.813454 29591 tablet_bootstrap.cc:492] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:50.813535 29591 ts_tablet_manager.cc:1367] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:50.813661 29592 raft_consensus.cc:348] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.813756 29592 raft_consensus.cc:374] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.813786 29592 raft_consensus.cc:729] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.813886 29592 consensus_queue.cc:260] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.813964 29591 raft_consensus.cc:348] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.814057 29591 raft_consensus.cc:374] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.814088 29591 raft_consensus.cc:729] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.814122 29592 ts_tablet_manager.cc:1398] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:50.814193 29591 consensus_queue.cc:260] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.814455 29591 ts_tablet_manager.cc:1398] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.829871 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:50.836189 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:50.837587 24836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.842303 24967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.844594 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:50.845361 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:50.849193 27274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:50.874887 24574 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.887444 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:50.893589 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 536a3171eab9403eab069fa86c70a8b5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.893748 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 536a3171eab9403eab069fa86c70a8b5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.894004 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536a3171eab9403eab069fa86c70a8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.894059 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536a3171eab9403eab069fa86c70a8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.896238 29607 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:50.896418 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 2c52abe3cfff44f88f41e1e885e5f2c9 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.896533 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 536a3171eab9403eab069fa86c70a8b5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.896672 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c52abe3cfff44f88f41e1e885e5f2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.896783 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536a3171eab9403eab069fa86c70a8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.897233 29608 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:50.897459 29607 tablet_bootstrap.cc:654] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.898453 29608 tablet_bootstrap.cc:654] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.898785 29610 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:50.899174 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 6662293cdf374e019f4197eca35ad8b6 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.899459 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6662293cdf374e019f4197eca35ad8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.899721 29610 tablet_bootstrap.cc:654] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.900270 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 6662293cdf374e019f4197eca35ad8b6 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.900501 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6662293cdf374e019f4197eca35ad8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.900734 29608 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:50.900815 29608 ts_tablet_manager.cc:1367] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:50.901209 29608 raft_consensus.cc:348] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:50.901312 29608 raft_consensus.cc:374] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.901340 29608 raft_consensus.cc:729] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.901482 29608 consensus_queue.cc:260] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:50.901589 29613 tablet_bootstrap.cc:492] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:50.901804 29608 ts_tablet_manager.cc:1398] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.902500 29613 tablet_bootstrap.cc:654] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.902652 29592 tablet_bootstrap.cc:492] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:50.903537 29592 tablet_bootstrap.cc:654] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.904625 29613 tablet_bootstrap.cc:492] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:50.904722 29613 ts_tablet_manager.cc:1367] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:50.905155 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 2c52abe3cfff44f88f41e1e885e5f2c9 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.905184 29613 raft_consensus.cc:348] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.905287 29613 raft_consensus.cc:374] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.905320 29613 raft_consensus.cc:729] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.905385 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c52abe3cfff44f88f41e1e885e5f2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.905326 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 2c52abe3cfff44f88f41e1e885e5f2c9 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.905429 29613 consensus_queue.cc:260] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.905614 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c52abe3cfff44f88f41e1e885e5f2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.905673 29613 ts_tablet_manager.cc:1398] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.907284 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 6662293cdf374e019f4197eca35ad8b6 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:50.907532 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6662293cdf374e019f4197eca35ad8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.909276 29591 tablet_bootstrap.cc:492] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:50.909412 29592 tablet_bootstrap.cc:492] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:50.909507 29592 ts_tablet_manager.cc:1367] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:50.909950 29592 raft_consensus.cc:348] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.910054 29592 raft_consensus.cc:374] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.910092 29592 raft_consensus.cc:729] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.910218 29591 tablet_bootstrap.cc:654] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.910209 29592 consensus_queue.cc:260] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.910498 29592 ts_tablet_manager.cc:1398] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:50.911466 29591 tablet_bootstrap.cc:492] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:50.911547 29591 ts_tablet_manager.cc:1367] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:50.912026 29591 raft_consensus.cc:348] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.912120 29591 raft_consensus.cc:374] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.912151 29591 raft_consensus.cc:729] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.912214 29607 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:50.912290 29607 ts_tablet_manager.cc:1367] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:56:50.912269 29591 consensus_queue.cc:260] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.912529 29591 ts_tablet_manager.cc:1398] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:50.912699 29591 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:50.912884 29607 raft_consensus.cc:348] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:50.912978 29607 raft_consensus.cc:374] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.913018 29607 raft_consensus.cc:729] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.913165 29607 consensus_queue.cc:260] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:50.914741 29607 ts_tablet_manager.cc:1398] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:50.914959 29607 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:50.915213 29591 tablet_bootstrap.cc:654] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.916009 29607 tablet_bootstrap.cc:654] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.916407 29591 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:50.916500 29591 ts_tablet_manager.cc:1367] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:50.916925 29591 raft_consensus.cc:348] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:50.917058 29591 raft_consensus.cc:374] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.917100 29591 raft_consensus.cc:729] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.917217 29591 consensus_queue.cc:260] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:50.917276 29610 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:50.917364 29610 ts_tablet_manager.cc:1367] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 19:56:50.917491 29591 ts_tablet_manager.cc:1398] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.917847 29610 raft_consensus.cc:348] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:50.917959 29610 raft_consensus.cc:374] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.917994 29610 raft_consensus.cc:729] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.918613 28709 raft_consensus.cc:482] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:50.918715 28709 raft_consensus.cc:504] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.919231 28709 leader_election.cc:288] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:50.919605 28856 consensus_queue.cc:1035] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.920040 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb47fb4ec4c4dfaa0fcf09f0a09d7a4" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:50.920186 24594 raft_consensus.cc:2454] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:50.920455 24660 leader_election.cc:302] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:50.920745 28709 raft_consensus.cc:2781] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:50.920810 28709 raft_consensus.cc:482] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:50.920852 28709 raft_consensus.cc:3037] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.921847 28709 raft_consensus.cc:504] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.922185 28709 leader_election.cc:288] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:50.922796 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb47fb4ec4c4dfaa0fcf09f0a09d7a4" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:50.922910 24594 raft_consensus.cc:3037] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.924006 24594 raft_consensus.cc:2454] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:50.924288 24660 leader_election.cc:302] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:50.924252 28856 consensus_queue.cc:1035] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:50.924435 28709 raft_consensus.cc:2781] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:50.924517 28709 raft_consensus.cc:686] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:50.924630 28709 consensus_queue.cc:237] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.925377 29610 consensus_queue.cc:260] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:50.926951 24452 catalog_manager.cc:5484] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:50.927799 29610 ts_tablet_manager.cc:1398] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:56:50.928007 29610 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:50.930205 29610 tablet_bootstrap.cc:654] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:50.931901 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb47fb4ec4c4dfaa0fcf09f0a09d7a4" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:50.932044 27162 raft_consensus.cc:2454] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:50.931922 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb47fb4ec4c4dfaa0fcf09f0a09d7a4" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:50.932161 27163 raft_consensus.cc:3037] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.933195 27163 raft_consensus.cc:2454] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:50.933917 29610 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:50.934355 29610 ts_tablet_manager.cc:1367] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:50.934824 29610 raft_consensus.cc:348] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:50.934978 29610 raft_consensus.cc:374] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.935060 29610 raft_consensus.cc:729] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.935185 29610 consensus_queue.cc:260] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:50.935956 29610 ts_tablet_manager.cc:1398] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:50.936627 29607 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:50.936705 29607 ts_tablet_manager.cc:1367] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 19:56:50.937130 29607 raft_consensus.cc:348] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:50.937230 29607 raft_consensus.cc:374] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:50.937258 29607 raft_consensus.cc:729] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:50.937386 29607 consensus_queue.cc:260] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:50.938819 29607 ts_tablet_manager.cc:1398] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:50.977144 26885 raft_consensus.cc:482] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:50.977263 26885 raft_consensus.cc:504] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.977699 26885 leader_election.cc:288] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:50.978873 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6662293cdf374e019f4197eca35ad8b6" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:50.979305 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6662293cdf374e019f4197eca35ad8b6" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:50.979432 27163 raft_consensus.cc:2454] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:50.979019 27032 raft_consensus.cc:2454] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:50.980005 24529 leader_election.cc:302] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:50.980650 26885 raft_consensus.cc:2781] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:50.980718 26885 raft_consensus.cc:482] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:50.980764 26885 raft_consensus.cc:3037] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.981853 26885 raft_consensus.cc:504] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.982174 26885 leader_election.cc:288] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:50.983358 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6662293cdf374e019f4197eca35ad8b6" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:50.983467 27163 raft_consensus.cc:3037] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.984478 27163 raft_consensus.cc:2454] T 6662293cdf374e019f4197eca35ad8b6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:50.984737 24531 leader_election.cc:302] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:50.984910 27943 raft_consensus.cc:2781] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:50.984989 27943 raft_consensus.cc:686] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:50.985085 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6662293cdf374e019f4197eca35ad8b6" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:50.985119 27943 consensus_queue.cc:237] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:50.985230 27032 raft_consensus.cc:3037] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:50.986222 27032 raft_consensus.cc:2454] T 6662293cdf374e019f4197eca35ad8b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:50.987323 24453 catalog_manager.cc:5484] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:51.032902 24447 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:51.035697 29519 raft_consensus.cc:482] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:51.035811 29519 raft_consensus.cc:504] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:51.036176 29519 leader_election.cc:288] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:51.036563 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c52abe3cfff44f88f41e1e885e5f2c9" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:51.036631 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c52abe3cfff44f88f41e1e885e5f2c9" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:51.036718 27163 raft_consensus.cc:2454] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:51.036754 27276 raft_consensus.cc:2454] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:51.036967 24594 raft_consensus.cc:1261] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:51.037045 24793 leader_election.cc:302] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:51.037230 27163 raft_consensus.cc:1261] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:51.037214 29519 raft_consensus.cc:2781] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:51.037277 29519 raft_consensus.cc:482] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:51.037313 29519 raft_consensus.cc:3037] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.037498 29500 consensus_queue.cc:1035] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.038338 29519 raft_consensus.cc:504] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:51.038663 29519 leader_election.cc:288] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:51.038970 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c52abe3cfff44f88f41e1e885e5f2c9" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:51.039129 27276 raft_consensus.cc:3037] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.039769 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c52abe3cfff44f88f41e1e885e5f2c9" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:51.039872 27163 raft_consensus.cc:3037] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.040072 27276 raft_consensus.cc:2454] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:51.040236 28709 consensus_queue.cc:1035] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.040323 24793 leader_election.cc:302] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:51.040463 29519 raft_consensus.cc:2781] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:51.040551 29519 raft_consensus.cc:686] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:51.040680 29519 consensus_queue.cc:237] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:51.040804 27163 raft_consensus.cc:2454] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:51.040979 29667 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:51.041344 29667 tablet.cc:1620] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:51.041529 29367 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:51.041824 29367 tablet.cc:1620] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:51.042708 24449 catalog_manager.cc:5484] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:51.044499 24453 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (table rebalancer_dynamic_table [id=f69515b0b8c547c0a4b5f0521f4a167a]). Expected version 1 got 0
I20240501 19:56:51.045429 24447 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (table rebalancer_dynamic_table [id=f69515b0b8c547c0a4b5f0521f4a167a]). Expected version 1 got 0
I20240501 19:56:51.047422 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=f69515b0b8c547c0a4b5f0521f4a167a] alter complete (version 1)
I20240501 19:56:51.048460 29368 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:51.048750 29368 tablet.cc:1620] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:51.049521 24452 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (table rebalancer_dynamic_table [id=f69515b0b8c547c0a4b5f0521f4a167a]). Expected version 1 got 0
I20240501 19:56:51.099381 29670 ts_tablet_manager.cc:902] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:51.100051 29670 tablet_copy_client.cc:308] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:56:51.100749 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet 51cd33a27a0a4754bbe044b0ff0ef262 from peer b9fd7c49d12b48a7ae73ab09876a0b82 ({username='slave'} at 127.17.65.4:45851)
I20240501 19:56:51.100880 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet 51cd33a27a0a4754bbe044b0ff0ef262 from peer b9fd7c49d12b48a7ae73ab09876a0b82 at {username='slave'} at 127.17.65.4:45851: session id = b9fd7c49d12b48a7ae73ab09876a0b82-51cd33a27a0a4754bbe044b0ff0ef262
I20240501 19:56:51.101991 27052 tablet_copy_source_session.cc:205] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.103559 29670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cd33a27a0a4754bbe044b0ff0ef262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.105480 29670 tablet_copy_client.cc:783] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.105686 29670 tablet_copy_client.cc:647] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.107188 29670 tablet_copy_client.cc:523] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.108569 29670 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:51.112000 29670 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.112231 29670 tablet_bootstrap.cc:492] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap complete.
I20240501 19:56:51.112383 29670 ts_tablet_manager.cc:1367] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:51.112789 29670 raft_consensus.cc:348] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.112907 29670 raft_consensus.cc:729] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.113101 29670 consensus_queue.cc:260] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.113346 29677 ts_tablet_manager.cc:902] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:51.113389 29670 ts_tablet_manager.cc:1398] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:51.113456 29676 ts_tablet_manager.cc:902] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:51.113956 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-51cd33a27a0a4754bbe044b0ff0ef262 received from {username='slave'} at 127.17.65.4:45851
I20240501 19:56:51.114042 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-51cd33a27a0a4754bbe044b0ff0ef262 on tablet 51cd33a27a0a4754bbe044b0ff0ef262 with peer b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:56:51.114118 29677 tablet_copy_client.cc:308] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:56:51.115473 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet 4851f59ff041424bae97a3ecaa9e191b from peer 9081292597be4a72a42b216fddcd7971 ({username='slave'} at 127.17.65.2:41119)
I20240501 19:56:51.115607 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet 4851f59ff041424bae97a3ecaa9e191b from peer 9081292597be4a72a42b216fddcd7971 at {username='slave'} at 127.17.65.2:41119: session id = 9081292597be4a72a42b216fddcd7971-4851f59ff041424bae97a3ecaa9e191b
I20240501 19:56:51.116734 27314 tablet_copy_source_session.cc:205] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.116771 29464 raft_consensus.cc:482] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:51.116866 29464 raft_consensus.cc:504] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:51.117221 29464 leader_election.cc:288] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:51.117444 29676 tablet_copy_client.cc:308] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:51.117756 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536a3171eab9403eab069fa86c70a8b5" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:51.117897 27425 raft_consensus.cc:2454] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:51.117884 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536a3171eab9403eab069fa86c70a8b5" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:51.118002 24856 raft_consensus.cc:2454] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:51.118036 29677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4851f59ff041424bae97a3ecaa9e191b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.118223 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet 164460e6fbc44fb1a9a12d52fea9254b from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:38719)
I20240501 19:56:51.118300 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet 164460e6fbc44fb1a9a12d52fea9254b from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:38719: session id = 53e34bb09d8e41129a5683079ff63610-164460e6fbc44fb1a9a12d52fea9254b
I20240501 19:56:51.118276 27228 leader_election.cc:302] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:51.118443 29464 raft_consensus.cc:2781] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:51.118512 29464 raft_consensus.cc:482] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:51.118546 29464 raft_consensus.cc:3037] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.119292 24746 tablet_copy_source_session.cc:205] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.119545 29464 raft_consensus.cc:504] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:51.119897 29464 leader_election.cc:288] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:51.120149 29676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164460e6fbc44fb1a9a12d52fea9254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.120270 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536a3171eab9403eab069fa86c70a8b5" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:51.120357 24856 raft_consensus.cc:3037] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.121096 24856 raft_consensus.cc:2454] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:51.121390 29677 tablet_copy_client.cc:783] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.121516 27228 leader_election.cc:302] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:51.121544 29677 tablet_copy_client.cc:647] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.121717 29464 raft_consensus.cc:2781] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:51.121806 29464 raft_consensus.cc:686] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:51.121935 29464 consensus_queue.cc:237] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:51.122290 29676 tablet_copy_client.cc:783] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.122437 29676 tablet_copy_client.cc:647] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.122970 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536a3171eab9403eab069fa86c70a8b5" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:51.123111 27425 raft_consensus.cc:3037] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.123721 24447 catalog_manager.cc:5484] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:51.124001 27425 raft_consensus.cc:2454] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:51.124511 29677 tablet_copy_client.cc:523] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.127007 29676 tablet_copy_client.cc:523] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.127149 29677 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:51.128127 29676 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:51.131686 29677 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.131979 29677 tablet_bootstrap.cc:492] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: Bootstrap complete.
I20240501 19:56:51.132179 29677 ts_tablet_manager.cc:1367] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:51.132622 29677 raft_consensus.cc:348] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.132784 29677 raft_consensus.cc:729] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.132938 29677 consensus_queue.cc:260] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.133222 29677 ts_tablet_manager.cc:1398] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.133885 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session 9081292597be4a72a42b216fddcd7971-4851f59ff041424bae97a3ecaa9e191b received from {username='slave'} at 127.17.65.2:41119
I20240501 19:56:51.133961 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session 9081292597be4a72a42b216fddcd7971-4851f59ff041424bae97a3ecaa9e191b on tablet 4851f59ff041424bae97a3ecaa9e191b with peer 9081292597be4a72a42b216fddcd7971
I20240501 19:56:51.152029 24447 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:51.152391 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:51.157011 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.161206 29667 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:51.161525 29667 tablet.cc:1620] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:51.161605 27274 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:51.163432 29367 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:51.163729 29367 tablet.cc:1620] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:51.167359 29368 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:51.167656 29368 tablet.cc:1620] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:51.170190 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=f69515b0b8c547c0a4b5f0521f4a167a] alter complete (version 2)
I20240501 19:56:51.173619 29676 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.174770 29676 tablet_bootstrap.cc:492] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:51.175153 29676 ts_tablet_manager.cc:1367] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.047s user 0.021s sys 0.004s
I20240501 19:56:51.177793 29676 raft_consensus.cc:348] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:51.178298 29676 raft_consensus.cc:729] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.178495 29676 consensus_queue.cc:260] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:51.178900 29676 ts_tablet_manager.cc:1398] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:51.179355 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-164460e6fbc44fb1a9a12d52fea9254b received from {username='slave'} at 127.17.65.7:38719
I20240501 19:56:51.179435 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-164460e6fbc44fb1a9a12d52fea9254b on tablet 164460e6fbc44fb1a9a12d52fea9254b with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:51.180876 29700 ts_tablet_manager.cc:902] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:51.181416 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.181418 29700 tablet_copy_client.cc:308] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:56:51.182201 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 92dd4bc514f146bb89083fe8a9315308 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:39423)
I20240501 19:56:51.182317 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 92dd4bc514f146bb89083fe8a9315308 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:39423: session id = 4a6e54e86a9949e5970b42e8e32e7561-92dd4bc514f146bb89083fe8a9315308
I20240501 19:56:51.183220 24878 tablet_copy_source_session.cc:205] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.185436 24705 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:51.186292 29700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92dd4bc514f146bb89083fe8a9315308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.194006 24574 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:51.194265 24967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.207204 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.214841 29700 tablet_copy_client.cc:783] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.215068 29700 tablet_copy_client.cc:647] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.231205 29700 tablet_copy_client.cc:523] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.232750 29700 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:51.237643 29705 ts_tablet_manager.cc:902] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:51.238168 29705 tablet_copy_client.cc:308] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:56:51.238605 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet 23ae3af247c1456e8b75640c25906faf from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:52411)
I20240501 19:56:51.238718 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet 23ae3af247c1456e8b75640c25906faf from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:52411: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-23ae3af247c1456e8b75640c25906faf
I20240501 19:56:51.239492 25009 tablet_copy_source_session.cc:205] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.240283 29705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ae3af247c1456e8b75640c25906faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.242431 29705 tablet_copy_client.cc:783] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.242574 29705 tablet_copy_client.cc:647] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.243692 29705 tablet_copy_client.cc:523] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.244921 29705 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:51.261359 29700 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.261662 29700 tablet_bootstrap.cc:492] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:56:51.261840 29700 ts_tablet_manager.cc:1367] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.003s
I20240501 19:56:51.262256 29700 raft_consensus.cc:348] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.262380 29700 raft_consensus.cc:729] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.262507 29700 consensus_queue.cc:260] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.262784 29700 ts_tablet_manager.cc:1398] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.263365 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-92dd4bc514f146bb89083fe8a9315308 received from {username='slave'} at 127.17.65.8:39423
I20240501 19:56:51.263463 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-92dd4bc514f146bb89083fe8a9315308 on tablet 92dd4bc514f146bb89083fe8a9315308 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:51.269441 29705 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.269685 29705 tablet_bootstrap.cc:492] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:56:51.269835 29705 ts_tablet_manager.cc:1367] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240501 19:56:51.270190 29705 raft_consensus.cc:348] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:51.270288 29705 raft_consensus.cc:729] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.270442 29705 consensus_queue.cc:260] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:51.270726 29705 ts_tablet_manager.cc:1398] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:51.271219 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-23ae3af247c1456e8b75640c25906faf received from {username='slave'} at 127.17.65.5:52411
I20240501 19:56:51.271301 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-23ae3af247c1456e8b75640c25906faf on tablet 23ae3af247c1456e8b75640c25906faf with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:51.276847 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:51.282475 29667 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:51.282963 29367 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:51.283600 29368 tablet.cc:1722] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:51.287488 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=f69515b0b8c547c0a4b5f0521f4a167a] alter complete (version 3)
I20240501 19:56:51.316175 29752 ts_tablet_manager.cc:902] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:51.316854 29752 tablet_copy_client.cc:308] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:56:51.317288 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 3b692b5b96534e79a32a8b06b92f32d6 from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:40849)
I20240501 19:56:51.317387 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 3b692b5b96534e79a32a8b06b92f32d6 from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:40849: session id = 29d1a658e2b946e0b412e5fef82db9d9-3b692b5b96534e79a32a8b06b92f32d6
I20240501 19:56:51.318356 24878 tablet_copy_source_session.cc:205] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.319950 29752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b692b5b96534e79a32a8b06b92f32d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.324607 29752 tablet_copy_client.cc:783] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.324772 29752 tablet_copy_client.cc:647] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.326781 29752 tablet_copy_client.cc:523] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.328431 29752 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:51.352948 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:51.358721 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.361703 29752 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.361997 29752 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:56:51.362162 29752 ts_tablet_manager.cc:1367] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.000s
I20240501 19:56:51.362571 29752 raft_consensus.cc:348] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:51.362694 29752 raft_consensus.cc:729] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.362812 29752 consensus_queue.cc:260] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:51.363109 29752 ts_tablet_manager.cc:1398] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:51.363591 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-3b692b5b96534e79a32a8b06b92f32d6 received from {username='slave'} at 127.17.65.6:40849
I20240501 19:56:51.363690 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-3b692b5b96534e79a32a8b06b92f32d6 on tablet 3b692b5b96534e79a32a8b06b92f32d6 with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:56:51.364243 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.372543 24574 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:51.376837 24967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.383893 27274 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:51.385923 24705 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:51.389864 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:51.390022 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:51.395377 27943 consensus_queue.cc:1035] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.396445 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4
I20240501 19:56:51.397362 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:51 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:51.397356 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:51 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:51.397622 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:51.397698 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.398164 29770 tablet_replica.cc:330] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:51.398417 29772 tablet_replica.cc:330] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:51.398627 29770 raft_consensus.cc:2229] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.398867 29772 raft_consensus.cc:2229] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.399997 26885 consensus_queue.cc:1035] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.400148 29772 raft_consensus.cc:2258] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.400298 29770 raft_consensus.cc:2258] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.401548 29772 ts_tablet_manager.cc:1875] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:51.402786 29770 ts_tablet_manager.cc:1875] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:51.418164 29770 ts_tablet_manager.cc:1888] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:51.418340 29770 log.cc:1196] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/1bb47fb4ec4c4dfaa0fcf09f0a09d7a4
I20240501 19:56:51.418736 29770 ts_tablet_manager.cc:1909] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:51.416013 29772 ts_tablet_manager.cc:1888] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:51.418954 29772 log.cc:1196] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/1bb47fb4ec4c4dfaa0fcf09f0a09d7a4
I20240501 19:56:51.419281 29772 ts_tablet_manager.cc:1909] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:51.420019 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (table rebalancer_dynamic_table_new_name [id=f69515b0b8c547c0a4b5f0521f4a167a]) successfully deleted
I20240501 19:56:51.420516 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (table rebalancer_dynamic_table_new_name [id=f69515b0b8c547c0a4b5f0521f4a167a]) successfully deleted
I20240501 19:56:51.436400 29771 tablet_replica.cc:330] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:51.436630 29771 raft_consensus.cc:2229] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:51.436862 29771 raft_consensus.cc:2258] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.437595 29771 ts_tablet_manager.cc:1875] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:51.439869 29771 ts_tablet_manager.cc:1888] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:51.439998 29771 log.cc:1196] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/1bb47fb4ec4c4dfaa0fcf09f0a09d7a4
I20240501 19:56:51.440367 29771 ts_tablet_manager.cc:1909] T 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:51.441040 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 1bb47fb4ec4c4dfaa0fcf09f0a09d7a4 (table rebalancer_dynamic_table_new_name [id=f69515b0b8c547c0a4b5f0521f4a167a]) successfully deleted
I20240501 19:56:51.450397 29519 consensus_queue.cc:1035] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.461148 28114 consensus_queue.cc:1035] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.498153 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:51.500316 29395 consensus_queue.cc:1035] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.503870 29464 consensus_queue.cc:1035] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.505831 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2a1675ab7d4111a50601295c0e7452 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa]), partition=
I20240501 19:56:51.506124 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2a1675ab7d4111a50601295c0e7452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.507347 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2a1675ab7d4111a50601295c0e7452 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa]), partition=
I20240501 19:56:51.507570 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2a1675ab7d4111a50601295c0e7452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.508208 29820 tablet_bootstrap.cc:492] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:51.509035 29820 tablet_bootstrap.cc:654] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.509488 29821 tablet_bootstrap.cc:492] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:51.509989 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2a1675ab7d4111a50601295c0e7452 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa]), partition=
I20240501 19:56:51.510236 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2a1675ab7d4111a50601295c0e7452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.511891 29820 tablet_bootstrap.cc:492] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:51.511970 29820 ts_tablet_manager.cc:1367] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:51.512460 29820 raft_consensus.cc:348] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.512558 29820 raft_consensus.cc:374] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.512593 29820 raft_consensus.cc:729] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.512702 29820 consensus_queue.cc:260] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.512966 29820 ts_tablet_manager.cc:1398] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.515264 29821 tablet_bootstrap.cc:654] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.517501 29823 tablet_bootstrap.cc:492] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:51.518530 29823 tablet_bootstrap.cc:654] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.519708 29821 tablet_bootstrap.cc:492] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:51.519826 29821 ts_tablet_manager.cc:1367] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:51.519841 29823 tablet_bootstrap.cc:492] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:51.519949 29823 ts_tablet_manager.cc:1367] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:51.520270 29821 raft_consensus.cc:348] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.520452 29823 raft_consensus.cc:348] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.520571 29823 raft_consensus.cc:374] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.520670 29823 raft_consensus.cc:729] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.520426 29821 raft_consensus.cc:374] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.520718 29821 raft_consensus.cc:729] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.520859 29823 consensus_queue.cc:260] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.521144 29821 consensus_queue.cc:260] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.521502 29821 ts_tablet_manager.cc:1398] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:51.529289 29823 ts_tablet_manager.cc:1398] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:51.544384 27406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.546304 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.551512 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:51.551566 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:51.552058 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.554652 24967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.556317 27274 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:51.585930 24574 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:51.641665 24727 raft_consensus.cc:1203] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:51.659737 27425 raft_consensus.cc:1203] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:51.660490 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:51.666922 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca876a9d51e4096955127e7038e2555 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.666937 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca876a9d51e4096955127e7038e2555 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.667265 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca876a9d51e4096955127e7038e2555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.667320 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca876a9d51e4096955127e7038e2555 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.667562 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca876a9d51e4096955127e7038e2555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.667601 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef4152fef8749f0994a952095225134 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.667896 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef4152fef8749f0994a952095225134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.669538 29877 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:51.669585 29820 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:51.669534 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef4152fef8749f0994a952095225134 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.669831 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef4152fef8749f0994a952095225134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.670516 29820 tablet_bootstrap.cc:654] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.671618 27274 tablet_service.cc:1467] Processing CreateTablet for tablet bcbe96677d2140d1aa22870197a9890f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.671777 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef4152fef8749f0994a952095225134 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.671869 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbe96677d2140d1aa22870197a9890f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.672022 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef4152fef8749f0994a952095225134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.672261 29878 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:51.673840 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca876a9d51e4096955127e7038e2555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.674407 27405 tablet_service.cc:1467] Processing CreateTablet for tablet bcbe96677d2140d1aa22870197a9890f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.674607 29877 tablet_bootstrap.cc:654] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.674614 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbe96677d2140d1aa22870197a9890f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.675431 29820 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:51.675518 29820 ts_tablet_manager.cc:1367] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:51.675760 29877 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:51.675841 29877 ts_tablet_manager.cc:1367] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:51.675992 29820 raft_consensus.cc:348] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.676088 29820 raft_consensus.cc:374] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.676117 29820 raft_consensus.cc:729] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.676234 29820 consensus_queue.cc:260] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.676295 29877 raft_consensus.cc:348] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.676410 29877 raft_consensus.cc:374] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.676448 29877 raft_consensus.cc:729] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.676539 29820 ts_tablet_manager.cc:1398] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.676573 29877 consensus_queue.cc:260] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.675016 29878 tablet_bootstrap.cc:654] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.676863 29877 ts_tablet_manager.cc:1398] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.677022 29877 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:51.677871 29877 tablet_bootstrap.cc:654] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.677959 24705 tablet_service.cc:1467] Processing CreateTablet for tablet bcbe96677d2140d1aa22870197a9890f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:51.678229 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbe96677d2140d1aa22870197a9890f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.679466 29882 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:51.680577 27276 raft_consensus.cc:1203] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:51.682147 29882 tablet_bootstrap.cc:654] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.682472 29821 tablet_bootstrap.cc:492] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:51.683470 29821 tablet_bootstrap.cc:654] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.683655 29882 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:51.683743 29882 ts_tablet_manager.cc:1367] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:51.684181 29882 raft_consensus.cc:348] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.684288 29882 raft_consensus.cc:374] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.684315 29882 raft_consensus.cc:729] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.684437 29882 consensus_queue.cc:260] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.684711 29882 ts_tablet_manager.cc:1398] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.687724 29877 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:51.687809 29877 ts_tablet_manager.cc:1367] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:51.688196 29880 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:51.688215 29877 raft_consensus.cc:348] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.688314 29877 raft_consensus.cc:374] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.688375 29877 raft_consensus.cc:729] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.688488 29877 consensus_queue.cc:260] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.688782 29877 ts_tablet_manager.cc:1398] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.689006 29880 tablet_bootstrap.cc:654] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.689980 29821 tablet_bootstrap.cc:492] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:51.690093 29821 ts_tablet_manager.cc:1367] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:51.690328 29878 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:51.690416 29878 ts_tablet_manager.cc:1367] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240501 19:56:51.690521 29821 raft_consensus.cc:348] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.690608 29821 raft_consensus.cc:374] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.690634 29821 raft_consensus.cc:729] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.690748 29821 consensus_queue.cc:260] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.690846 29878 raft_consensus.cc:348] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.690933 29878 raft_consensus.cc:374] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.690961 29878 raft_consensus.cc:729] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.691011 29821 ts_tablet_manager.cc:1398] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.691124 29878 consensus_queue.cc:260] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.691391 29878 ts_tablet_manager.cc:1398] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:51.691848 29878 tablet_bootstrap.cc:492] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:51.695683 29878 tablet_bootstrap.cc:654] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.698235 29880 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:51.698619 29880 ts_tablet_manager.cc:1367] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:51.699187 29880 raft_consensus.cc:348] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.699637 29880 raft_consensus.cc:374] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.699728 29880 raft_consensus.cc:729] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.699913 29880 consensus_queue.cc:260] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.700448 29878 tablet_bootstrap.cc:492] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:51.700531 29878 ts_tablet_manager.cc:1367] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 19:56:51.701004 29880 ts_tablet_manager.cc:1398] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:51.700940 29878 raft_consensus.cc:348] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.701054 29878 raft_consensus.cc:374] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.701093 29878 raft_consensus.cc:729] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.701246 29878 consensus_queue.cc:260] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.701290 29880 tablet_bootstrap.cc:492] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:51.701505 29878 ts_tablet_manager.cc:1398] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:51.702131 29880 tablet_bootstrap.cc:654] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.704566 29880 tablet_bootstrap.cc:492] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:51.704697 29880 ts_tablet_manager.cc:1367] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:51.705322 29880 raft_consensus.cc:348] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.705420 29880 raft_consensus.cc:374] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.705454 29880 raft_consensus.cc:729] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.705590 29880 consensus_queue.cc:260] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.705947 29880 ts_tablet_manager.cc:1398] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.721210 24987 raft_consensus.cc:1203] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:51.735893 29464 raft_consensus.cc:482] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:51.736020 29464 raft_consensus.cc:504] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.736428 29464 leader_election.cc:288] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:51.736757 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef4152fef8749f0994a952095225134" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:51.736891 24987 raft_consensus.cc:2454] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:51.737155 27228 leader_election.cc:302] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:51.737306 29464 raft_consensus.cc:2781] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:51.737360 29464 raft_consensus.cc:482] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:51.737392 29464 raft_consensus.cc:3037] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.737543 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef4152fef8749f0994a952095225134" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:51.737684 27425 raft_consensus.cc:2454] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:51.738389 29464 raft_consensus.cc:504] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.738714 29464 leader_election.cc:288] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:51.738982 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef4152fef8749f0994a952095225134" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:51.739002 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef4152fef8749f0994a952095225134" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:51.739101 24987 raft_consensus.cc:3037] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.739169 27425 raft_consensus.cc:3037] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.740067 24987 raft_consensus.cc:2454] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:51.740108 27425 raft_consensus.cc:2454] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:51.740392 27229 leader_election.cc:302] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:51.740598 29464 raft_consensus.cc:2781] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:51.740692 29464 raft_consensus.cc:686] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:51.740821 29464 consensus_queue.cc:237] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:51.742516 24450 catalog_manager.cc:5484] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:51.752508 29626 raft_consensus.cc:482] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:51.752619 29626 raft_consensus.cc:504] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.753021 29626 leader_election.cc:288] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:51.753353 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2a1675ab7d4111a50601295c0e7452" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:51.753402 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2a1675ab7d4111a50601295c0e7452" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:51.753504 27032 raft_consensus.cc:2454] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:51.753525 24727 raft_consensus.cc:2454] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:51.753845 27096 leader_election.cc:302] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:51.753978 29626 raft_consensus.cc:2781] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:51.754031 29626 raft_consensus.cc:482] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:51.754070 29626 raft_consensus.cc:3037] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.755132 29626 raft_consensus.cc:504] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.755471 29626 leader_election.cc:288] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:51.755764 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2a1675ab7d4111a50601295c0e7452" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:51.755887 27032 raft_consensus.cc:3037] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.755894 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2a1675ab7d4111a50601295c0e7452" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:51.755997 24727 raft_consensus.cc:3037] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.756853 27032 raft_consensus.cc:2454] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:51.756987 24727 raft_consensus.cc:2454] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:51.757292 27096 leader_election.cc:302] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:51.757561 29626 raft_consensus.cc:2781] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:51.757656 29626 raft_consensus.cc:686] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:51.757776 29626 consensus_queue.cc:237] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.759442 24450 catalog_manager.cc:5484] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:51.775945 29334 raft_consensus.cc:482] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:51.776053 29334 raft_consensus.cc:504] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.776405 29334 leader_election.cc:288] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:51.776726 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe96677d2140d1aa22870197a9890f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:51.776854 27278 raft_consensus.cc:2454] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:51.777105 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe96677d2140d1aa22870197a9890f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:51.777225 24727 raft_consensus.cc:2454] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:51.778059 27361 leader_election.cc:302] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:51.778424 29334 raft_consensus.cc:2781] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:51.778491 29334 raft_consensus.cc:482] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:51.778532 29334 raft_consensus.cc:3037] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.779592 29334 raft_consensus.cc:504] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.779946 29334 leader_election.cc:288] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:51.780141 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe96677d2140d1aa22870197a9890f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:51.780244 27278 raft_consensus.cc:3037] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.780699 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe96677d2140d1aa22870197a9890f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:51.780812 24727 raft_consensus.cc:3037] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.781267 27278 raft_consensus.cc:2454] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:51.782609 24727 raft_consensus.cc:2454] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:51.782939 27359 leader_election.cc:302] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:51.783169 29334 raft_consensus.cc:2781] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:51.783305 29334 raft_consensus.cc:686] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:51.783455 29334 consensus_queue.cc:237] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:51.785022 27142 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.788210 24450 catalog_manager.cc:5484] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:51.794728 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.797048 27405 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:51.797547 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.798388 27274 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.801071 24966 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.803020 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:51.809635 24574 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:51.835517 27032 raft_consensus.cc:1203] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:51.865145 27032 consensus_queue.cc:237] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.866305 24987 raft_consensus.cc:1261] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:51.866417 24594 raft_consensus.cc:1261] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:51.866766 29922 consensus_queue.cc:1035] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:51.866890 29384 consensus_queue.cc:1035] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:51.868579 29384 raft_consensus.cc:2932] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:51.871106 24450 catalog_manager.cc:5484] T 51cd33a27a0a4754bbe044b0ff0ef262 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:51.871280 24987 raft_consensus.cc:2932] T 51cd33a27a0a4754bbe044b0ff0ef262 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:51.871280 24594 raft_consensus.cc:2932] T 51cd33a27a0a4754bbe044b0ff0ef262 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:51.873922 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cd33a27a0a4754bbe044b0ff0ef262 with delete_type TABLET_DATA_TOMBSTONED (TS 29d1a658e2b946e0b412e5fef82db9d9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:51.874183 29772 tablet_replica.cc:330] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:51.874334 29772 raft_consensus.cc:2229] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.874502 29772 raft_consensus.cc:2258] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.874951 29772 ts_tablet_manager.cc:1875] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.877161 29772 ts_tablet_manager.cc:1888] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:56:51.877241 29772 log.cc:1196] T 51cd33a27a0a4754bbe044b0ff0ef262 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/51cd33a27a0a4754bbe044b0ff0ef262
I20240501 19:56:51.877737 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 51cd33a27a0a4754bbe044b0ff0ef262 (table rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]) successfully deleted
I20240501 19:56:51.908000 27162 raft_consensus.cc:1203] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:51.919798 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:51.925663 24727 raft_consensus.cc:1261] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:51.926254 29626 consensus_queue.cc:1035] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.927495 27032 raft_consensus.cc:1261] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:51.928282 29626 consensus_queue.cc:1035] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:51.930585 29966 mvcc.cc:205] Tried to move back new op lower bound from 7022974615242375168 to 7022974614560944128. Current Snapshot: MvccSnapshot[applied={T|T < 7022974615242375168}]
I20240501 19:56:51.930711 24454 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 5f2a1675ab7d4111a50601295c0e7452 (table rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa]). Expected version 1 got 0
I20240501 19:56:51.931010 29969 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:51.932346 24449 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 5f2a1675ab7d4111a50601295c0e7452 (table rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa]). Expected version 1 got 0
I20240501 19:56:51.932885 29969 tablet.cc:1620] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:51.933072 29971 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:51.933517 24449 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 5f2a1675ab7d4111a50601295c0e7452 (table rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa]). Expected version 1 got 0
I20240501 19:56:51.933750 29971 tablet.cc:1620] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:51.938768 29972 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:51.939191 29972 tablet.cc:1620] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:51.939523 24445 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 5f2a1675ab7d4111a50601295c0e7452 (table rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa]). Expected version 1 got 0
I20240501 19:56:51.948019 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa] alter complete (version 1)
I20240501 19:56:51.986426 26885 raft_consensus.cc:482] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:51.986557 26885 raft_consensus.cc:504] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.986956 26885 leader_election.cc:288] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:51.987288 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca876a9d51e4096955127e7038e2555" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:51.987397 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca876a9d51e4096955127e7038e2555" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:51.987455 27032 raft_consensus.cc:2454] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:51.987515 24987 raft_consensus.cc:2454] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:51.987802 24529 leader_election.cc:302] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:51.987969 26885 raft_consensus.cc:2781] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:51.988018 26885 raft_consensus.cc:482] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:51.988059 26885 raft_consensus.cc:3037] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.989038 26885 raft_consensus.cc:504] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.989326 26885 leader_election.cc:288] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:51.989611 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca876a9d51e4096955127e7038e2555" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:51.989629 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca876a9d51e4096955127e7038e2555" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:51.989723 27032 raft_consensus.cc:3037] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.989728 24987 raft_consensus.cc:3037] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.990628 24987 raft_consensus.cc:2454] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:51.990680 27032 raft_consensus.cc:2454] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:51.990944 24529 leader_election.cc:302] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:51.991170 26885 raft_consensus.cc:2781] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:51.991356 26885 raft_consensus.cc:686] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:51.991508 26885 consensus_queue.cc:237] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:51.993113 24445 catalog_manager.cc:5484] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:52.010411 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:52.011831 27142 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.015103 24966 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.017314 27274 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.019774 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:52.022279 27405 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:52.050875 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.052947 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:52.062189 29969 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:52.062528 29969 tablet.cc:1620] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:52.062525 29972 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:52.062637 29971 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:52.062826 29972 tablet.cc:1620] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:52.062891 29971 tablet.cc:1620] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:52.069568 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:52.069896 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=535c96cc1d9c4365babca4305ed95dfa] alter complete (version 2)
I20240501 19:56:52.108541 24594 consensus_queue.cc:237] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.109825 27032 raft_consensus.cc:1261] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.109925 24856 raft_consensus.cc:1261] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.110251 29973 consensus_queue.cc:1035] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.110460 26885 consensus_queue.cc:1035] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.112247 30004 raft_consensus.cc:2932] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:52.112591 24856 raft_consensus.cc:2932] T e996e96ce69348c09adc99dc7d9c7125 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:52.112598 27032 raft_consensus.cc:2932] T e996e96ce69348c09adc99dc7d9c7125 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:52.115339 24445 catalog_manager.cc:5484] T e996e96ce69348c09adc99dc7d9c7125 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:52.120179 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet e996e96ce69348c09adc99dc7d9c7125 with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:52.120508 30010 tablet_replica.cc:330] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:52.121024 30010 raft_consensus.cc:2229] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.121186 30010 raft_consensus.cc:2258] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.121702 30010 ts_tablet_manager.cc:1875] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.124171 30010 ts_tablet_manager.cc:1888] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:52.124231 30010 log.cc:1196] T e996e96ce69348c09adc99dc7d9c7125 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/e996e96ce69348c09adc99dc7d9c7125
I20240501 19:56:52.124701 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet e996e96ce69348c09adc99dc7d9c7125 (table rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]) successfully deleted
I20240501 19:56:52.174762 24447 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:52.180578 29969 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:52.181649 29971 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:52.183643 29972 tablet.cc:1722] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:52.188344 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=535c96cc1d9c4365babca4305ed95dfa] alter complete (version 3)
I20240501 19:56:52.205371 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:52.205371 27142 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.209379 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:52.217352 27274 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.226814 27405 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:52.238842 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.243903 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:52.254616 24966 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.291494 24447 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:52.291658 24447 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:52.293790 24447 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 5f2a1675ab7d4111a50601295c0e7452
I20240501 19:56:52.294461 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2a1675ab7d4111a50601295c0e7452 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:52 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:52.294469 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2a1675ab7d4111a50601295c0e7452 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:52 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:52.294621 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2a1675ab7d4111a50601295c0e7452 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:52.294878 29772 tablet_replica.cc:330] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:52.294895 30074 tablet_replica.cc:330] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:52.295017 30074 raft_consensus.cc:2229] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.295017 29772 raft_consensus.cc:2229] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:52.295192 30074 raft_consensus.cc:2258] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.295276 30075 tablet_replica.cc:330] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:52.295339 29772 raft_consensus.cc:2258] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.295413 30075 raft_consensus.cc:2229] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.295522 30075 raft_consensus.cc:2258] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.295549 24594 consensus_queue.cc:237] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:52.295864 29772 ts_tablet_manager.cc:1875] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:52.296614 30074 ts_tablet_manager.cc:1875] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:52.296862 30075 ts_tablet_manager.cc:1875] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:52.297909 27425 raft_consensus.cc:1261] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.298012 29772 ts_tablet_manager.cc:1888] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:52.298213 29772 log.cc:1196] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/5f2a1675ab7d4111a50601295c0e7452
I20240501 19:56:52.298281 30004 consensus_queue.cc:1035] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.298580 29772 ts_tablet_manager.cc:1909] T 5f2a1675ab7d4111a50601295c0e7452 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:52.299093 30074 ts_tablet_manager.cc:1888] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:52.299168 30074 log.cc:1196] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/5f2a1675ab7d4111a50601295c0e7452
I20240501 19:56:52.299420 30075 ts_tablet_manager.cc:1888] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:52.299513 30074 ts_tablet_manager.cc:1909] T 5f2a1675ab7d4111a50601295c0e7452 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:52.299499 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 5f2a1675ab7d4111a50601295c0e7452 (table rebalancer_dynamic_table_new_name [id=535c96cc1d9c4365babca4305ed95dfa]) successfully deleted
I20240501 19:56:52.299538 30075 log.cc:1196] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/5f2a1675ab7d4111a50601295c0e7452
I20240501 19:56:52.299886 30075 ts_tablet_manager.cc:1909] T 5f2a1675ab7d4111a50601295c0e7452 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:52.300390 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 5f2a1675ab7d4111a50601295c0e7452 (table rebalancer_dynamic_table_new_name [id=535c96cc1d9c4365babca4305ed95dfa]) successfully deleted
I20240501 19:56:52.300715 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 5f2a1675ab7d4111a50601295c0e7452 (table rebalancer_dynamic_table_new_name [id=535c96cc1d9c4365babca4305ed95dfa]) successfully deleted
I20240501 19:56:52.301198 24725 raft_consensus.cc:1261] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.301308 30004 raft_consensus.cc:2932] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:52.301723 26885 consensus_queue.cc:1035] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.302923 24725 raft_consensus.cc:2932] T f89e81297b66402ea09bdb415fb7f385 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:52.303431 27425 raft_consensus.cc:2932] T f89e81297b66402ea09bdb415fb7f385 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:52.304080 24447 catalog_manager.cc:5484] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:52.307009 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet f89e81297b66402ea09bdb415fb7f385 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:52.307370 30079 tablet_replica.cc:330] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:52.307560 30079 raft_consensus.cc:2229] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.307721 30079 raft_consensus.cc:2258] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.308162 30079 ts_tablet_manager.cc:1875] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.310479 30079 ts_tablet_manager.cc:1888] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:52.310593 30079 log.cc:1196] T f89e81297b66402ea09bdb415fb7f385 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/f89e81297b66402ea09bdb415fb7f385
I20240501 19:56:52.310583 29464 consensus_queue.cc:1035] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:52.311095 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet f89e81297b66402ea09bdb415fb7f385 (table rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]) successfully deleted
I20240501 19:56:52.314318 29464 consensus_queue.cc:1035] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:52.366122 29334 consensus_queue.cc:1035] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:52.371430 29334 consensus_queue.cc:1035] T bcbe96677d2140d1aa22870197a9890f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:52.388700 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:52.389603 27405 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:52.396433 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:52.396900 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:52.407285 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:52.415863 27405 tablet_service.cc:1467] Processing CreateTablet for tablet c783f332669845cf93df04cfed4bee0f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d50b80e5ee3649468d515f2d0b4a0344]), partition=
I20240501 19:56:52.416121 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c783f332669845cf93df04cfed4bee0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.418656 24574 tablet_service.cc:1467] Processing CreateTablet for tablet c783f332669845cf93df04cfed4bee0f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d50b80e5ee3649468d515f2d0b4a0344]), partition=
I20240501 19:56:52.418896 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c783f332669845cf93df04cfed4bee0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.419864 27011 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:52.421483 27012 tablet_service.cc:1467] Processing CreateTablet for tablet c783f332669845cf93df04cfed4bee0f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d50b80e5ee3649468d515f2d0b4a0344]), partition=
I20240501 19:56:52.421700 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c783f332669845cf93df04cfed4bee0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.425431 30147 tablet_bootstrap.cc:492] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:52.427210 30146 tablet_bootstrap.cc:492] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:52.428519 30147 tablet_bootstrap.cc:654] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.429875 27274 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.430310 30146 tablet_bootstrap.cc:654] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.436884 30147 tablet_bootstrap.cc:492] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:52.437242 30147 ts_tablet_manager.cc:1367] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:52.437440 24966 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.449066 24836 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.451439 30147 raft_consensus.cc:348] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.451560 30147 raft_consensus.cc:374] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.451589 30147 raft_consensus.cc:729] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.451707 30147 consensus_queue.cc:260] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.452545 30147 ts_tablet_manager.cc:1398] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 19:56:52.452888 30145 tablet_bootstrap.cc:492] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:52.454110 30145 tablet_bootstrap.cc:654] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.463330 30145 tablet_bootstrap.cc:492] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:52.463430 30145 ts_tablet_manager.cc:1367] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:52.463637 30146 tablet_bootstrap.cc:492] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:52.463738 30146 ts_tablet_manager.cc:1367] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.001s
I20240501 19:56:52.463889 30145 raft_consensus.cc:348] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.464011 30145 raft_consensus.cc:374] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.464053 30145 raft_consensus.cc:729] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.464227 30145 consensus_queue.cc:260] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.464381 30146 raft_consensus.cc:348] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.464512 30146 raft_consensus.cc:374] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.464555 30146 raft_consensus.cc:729] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.464682 30146 consensus_queue.cc:260] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.464943 30146 ts_tablet_manager.cc:1398] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:52.465010 30145 ts_tablet_manager.cc:1398] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:52.496109 30132 raft_consensus.cc:482] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:52.496202 30132 raft_consensus.cc:504] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.496659 30132 leader_election.cc:288] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:52.496946 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c783f332669845cf93df04cfed4bee0f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:52.497072 24595 raft_consensus.cc:2454] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:52.497139 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c783f332669845cf93df04cfed4bee0f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:52.497262 27032 raft_consensus.cc:2454] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:52.497319 27425 consensus_queue.cc:237] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:52.497390 27359 leader_election.cc:302] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:52.497684 30132 raft_consensus.cc:2781] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:52.497771 30132 raft_consensus.cc:482] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:52.497813 30132 raft_consensus.cc:3037] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.498891 30132 raft_consensus.cc:504] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.498917 24857 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:52.498961 24987 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:52.499186 30132 leader_election.cc:288] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:52.499456 30132 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.499630 30169 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.499752 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c783f332669845cf93df04cfed4bee0f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:52.499840 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c783f332669845cf93df04cfed4bee0f" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:52.499876 27032 raft_consensus.cc:3037] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.499945 24595 raft_consensus.cc:3037] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.500869 27032 raft_consensus.cc:2454] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:52.500908 24595 raft_consensus.cc:2454] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:52.501241 27359 leader_election.cc:302] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:52.501381 30169 raft_consensus.cc:2781] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:52.501291 30132 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:52.501446 30169 raft_consensus.cc:686] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:52.501583 30169 consensus_queue.cc:237] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.502660 24857 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:52.503330 24448 catalog_manager.cc:5484] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:52.503618 24987 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:52.506502 24453 catalog_manager.cc:5484] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:52.509471 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 433347ce828b4d17a70308c421ab4f1f with delete_type TABLET_DATA_TOMBSTONED (TS 9081292597be4a72a42b216fddcd7971 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:52.509766 30074 tablet_replica.cc:330] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:52.509927 30074 raft_consensus.cc:2229] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.510068 30074 raft_consensus.cc:2258] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.510601 30074 ts_tablet_manager.cc:1875] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.512389 30074 ts_tablet_manager.cc:1888] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:56:52.512455 30074 log.cc:1196] T 433347ce828b4d17a70308c421ab4f1f P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/433347ce828b4d17a70308c421ab4f1f
I20240501 19:56:52.512933 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 433347ce828b4d17a70308c421ab4f1f (table rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]) successfully deleted
I20240501 19:56:52.562513 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:52.571434 30004 consensus_queue.cc:1035] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:52.571728 24705 tablet_service.cc:1467] Processing CreateTablet for tablet d6227a1f28614fbcb1b5a573d125e670 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.572011 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6227a1f28614fbcb1b5a573d125e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.572113 27274 tablet_service.cc:1467] Processing CreateTablet for tablet d6227a1f28614fbcb1b5a573d125e670 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.572387 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6227a1f28614fbcb1b5a573d125e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.574362 30220 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:52.574481 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 30d30871b80a4baa8c10489b752b0b63 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.574750 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 7959d3c11a6d4c7485c8e030e1d5092b (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.575466 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 30d30871b80a4baa8c10489b752b0b63 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.575749 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d30871b80a4baa8c10489b752b0b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.575837 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 7959d3c11a6d4c7485c8e030e1d5092b (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.576066 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7959d3c11a6d4c7485c8e030e1d5092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.576267 27012 tablet_service.cc:1467] Processing CreateTablet for tablet d6227a1f28614fbcb1b5a573d125e670 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.576520 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6227a1f28614fbcb1b5a573d125e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.576808 30220 tablet_bootstrap.cc:654] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.579139 30220 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:52.579216 30147 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:52.579269 30220 ts_tablet_manager.cc:1367] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:52.579694 30220 raft_consensus.cc:348] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.579784 30220 raft_consensus.cc:374] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.579813 30220 raft_consensus.cc:729] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.579926 30220 consensus_queue.cc:260] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.580068 30147 tablet_bootstrap.cc:654] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.580312 30220 ts_tablet_manager.cc:1398] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.580451 30220 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:52.581393 30220 tablet_bootstrap.cc:654] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.585144 30224 tablet_bootstrap.cc:492] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:52.585986 30224 tablet_bootstrap.cc:654] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.586184 30221 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:52.574760 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d30871b80a4baa8c10489b752b0b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.587023 29973 consensus_queue.cc:1035] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:52.588891 30221 tablet_bootstrap.cc:654] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.588908 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7959d3c11a6d4c7485c8e030e1d5092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.594244 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 7959d3c11a6d4c7485c8e030e1d5092b (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.594496 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7959d3c11a6d4c7485c8e030e1d5092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.596347 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 30d30871b80a4baa8c10489b752b0b63 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.596699 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d30871b80a4baa8c10489b752b0b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.601465 30230 tablet_bootstrap.cc:492] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:52.603576 30147 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:52.603617 30221 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:52.603668 30147 ts_tablet_manager.cc:1367] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 19:56:52.603693 30221 ts_tablet_manager.cc:1367] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 19:56:52.604061 30147 raft_consensus.cc:348] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.604144 30147 raft_consensus.cc:374] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.604169 30147 raft_consensus.cc:729] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.604123 30221 raft_consensus.cc:348] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.604224 30221 raft_consensus.cc:374] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.604262 30221 raft_consensus.cc:729] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.604277 30147 consensus_queue.cc:260] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.604367 30221 consensus_queue.cc:260] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.604525 30147 ts_tablet_manager.cc:1398] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.604647 30147 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:52.604708 30221 ts_tablet_manager.cc:1398] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.604843 30221 tablet_bootstrap.cc:492] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:52.605401 30147 tablet_bootstrap.cc:654] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.605669 30221 tablet_bootstrap.cc:654] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.607473 30221 tablet_bootstrap.cc:492] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:52.607553 30221 ts_tablet_manager.cc:1367] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:52.607985 30221 raft_consensus.cc:348] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:52.608099 30221 raft_consensus.cc:374] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.608139 30221 raft_consensus.cc:729] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.608273 30221 consensus_queue.cc:260] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:52.608528 30221 ts_tablet_manager.cc:1398] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.616041 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:52.620039 24595 raft_consensus.cc:1261] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:52.620460 30173 consensus_queue.cc:1035] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:52.621062 27032 raft_consensus.cc:1261] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:52.621491 30173 consensus_queue.cc:1035] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:52.623205 24452 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet c783f332669845cf93df04cfed4bee0f (table rebalancer_dynamic_table [id=d50b80e5ee3649468d515f2d0b4a0344]). Expected version 1 got 0
I20240501 19:56:52.624600 24452 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet c783f332669845cf93df04cfed4bee0f (table rebalancer_dynamic_table [id=d50b80e5ee3649468d515f2d0b4a0344]). Expected version 1 got 0
I20240501 19:56:52.626312 29972 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:52.626653 29972 tablet.cc:1620] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:52.628719 24450 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet c783f332669845cf93df04cfed4bee0f (table rebalancer_dynamic_table [id=d50b80e5ee3649468d515f2d0b4a0344]). Expected version 1 got 0
I20240501 19:56:52.628851 30246 mvcc.cc:205] Tried to move back new op lower bound from 7022974618086866944 to 7022974617607561216. Current Snapshot: MvccSnapshot[applied={T|T < 7022974618086866944}]
I20240501 19:56:52.630218 24705 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:52.630796 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:52.624840 30249 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:52.631733 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=d50b80e5ee3649468d515f2d0b4a0344] alter complete (version 1)
I20240501 19:56:52.631987 30249 tablet.cc:1620] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:52.632328 30250 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:52.632691 30250 tablet.cc:1620] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:52.603025 30230 tablet_bootstrap.cc:654] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.642554 27405 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:52.642899 27274 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.645596 30224 tablet_bootstrap.cc:492] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:52.645717 30224 ts_tablet_manager.cc:1367] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.000s
I20240501 19:56:52.646221 30224 raft_consensus.cc:348] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:52.646343 30224 raft_consensus.cc:374] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.646378 30224 raft_consensus.cc:729] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.646528 30224 consensus_queue.cc:260] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:52.646886 30224 ts_tablet_manager.cc:1398] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.646898 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:52.649974 30147 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:52.650059 30147 ts_tablet_manager.cc:1367] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20240501 19:56:52.650472 30147 raft_consensus.cc:348] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.650578 30147 raft_consensus.cc:374] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.650606 30147 raft_consensus.cc:729] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.650725 30147 consensus_queue.cc:260] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.650979 30147 ts_tablet_manager.cc:1398] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.655467 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.656406 30220 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:52.656515 30220 ts_tablet_manager.cc:1367] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.000s
I20240501 19:56:52.657109 30220 raft_consensus.cc:348] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.657208 30220 raft_consensus.cc:374] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.657235 30220 raft_consensus.cc:729] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.657341 30220 consensus_queue.cc:260] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.657572 30220 ts_tablet_manager.cc:1398] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.661801 24966 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.665833 30230 tablet_bootstrap.cc:492] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:52.665975 30230 ts_tablet_manager.cc:1367] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.000s
I20240501 19:56:52.666440 30230 raft_consensus.cc:348] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:52.666694 30230 raft_consensus.cc:374] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.666792 30230 raft_consensus.cc:729] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.666960 30230 consensus_queue.cc:260] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:52.667325 30230 ts_tablet_manager.cc:1398] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.667596 30230 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:52.669354 30230 tablet_bootstrap.cc:654] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.671880 30230 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:52.672008 30230 ts_tablet_manager.cc:1367] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:52.672441 30230 raft_consensus.cc:348] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.672704 30230 raft_consensus.cc:374] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.672778 30230 raft_consensus.cc:729] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.672953 30230 consensus_queue.cc:260] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.673902 30230 ts_tablet_manager.cc:1398] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:52.691304 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:52.708391 28709 raft_consensus.cc:482] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:52.708539 28709 raft_consensus.cc:504] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.708987 28709 leader_election.cc:288] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:52.709549 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:52.709692 27032 raft_consensus.cc:2454] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:52.709769 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:52.709921 24857 raft_consensus.cc:2454] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:52.710011 24660 leader_election.cc:302] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:52.710211 28709 raft_consensus.cc:2781] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:52.710264 28709 raft_consensus.cc:482] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:52.710299 28709 raft_consensus.cc:3037] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.711526 28709 raft_consensus.cc:504] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.711829 28709 leader_election.cc:288] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:52.712152 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:52.712257 27032 raft_consensus.cc:3037] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.712289 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:52.712396 24857 raft_consensus.cc:3037] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.713290 27032 raft_consensus.cc:2454] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:52.713302 24857 raft_consensus.cc:2454] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:52.713557 24661 leader_election.cc:302] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:52.713735 28709 raft_consensus.cc:2781] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:52.713804 28709 raft_consensus.cc:686] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:52.713937 28709 consensus_queue.cc:237] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.717094 24450 catalog_manager.cc:5484] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:52.731518 27278 consensus_queue.cc:237] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:52.732867 24725 raft_consensus.cc:1261] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:52.732954 27032 raft_consensus.cc:1261] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:52.733237 29464 consensus_queue.cc:1035] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:52.733466 30085 consensus_queue.cc:1035] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:52.735298 30272 raft_consensus.cc:2932] T 4851f59ff041424bae97a3ecaa9e191b P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:52.735661 24725 raft_consensus.cc:2932] T 4851f59ff041424bae97a3ecaa9e191b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:52.735700 27032 raft_consensus.cc:2932] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:52.738179 24450 catalog_manager.cc:5484] T 4851f59ff041424bae97a3ecaa9e191b P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:52.738886 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:52.742785 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 4851f59ff041424bae97a3ecaa9e191b with delete_type TABLET_DATA_TOMBSTONED (TS 29d1a658e2b946e0b412e5fef82db9d9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:52.743016 29772 tablet_replica.cc:330] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:52.743196 29772 raft_consensus.cc:2229] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.743338 29772 raft_consensus.cc:2258] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.743814 29772 ts_tablet_manager.cc:1875] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.745988 29772 ts_tablet_manager.cc:1888] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:56:52.746053 29772 log.cc:1196] T 4851f59ff041424bae97a3ecaa9e191b P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/4851f59ff041424bae97a3ecaa9e191b
I20240501 19:56:52.746448 30249 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:52.746812 30249 tablet.cc:1620] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:52.747740 29972 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:52.747996 29972 tablet.cc:1620] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:52.748831 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 4851f59ff041424bae97a3ecaa9e191b (table rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]) successfully deleted
I20240501 19:56:52.749008 30250 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:52.749315 30250 tablet.cc:1620] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:52.751545 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=d50b80e5ee3649468d515f2d0b4a0344] alter complete (version 2)
I20240501 19:56:52.772810 28709 raft_consensus.cc:482] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:52.772907 28709 raft_consensus.cc:504] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.773222 28709 leader_election.cc:288] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:52.774004 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6227a1f28614fbcb1b5a573d125e670" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:52.774128 27278 raft_consensus.cc:2454] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:52.774333 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6227a1f28614fbcb1b5a573d125e670" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:52.774456 27032 raft_consensus.cc:2454] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:52.774912 24660 leader_election.cc:302] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:52.775310 28709 raft_consensus.cc:2781] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:52.775360 28709 raft_consensus.cc:482] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:52.775395 28709 raft_consensus.cc:3037] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.776427 28709 raft_consensus.cc:504] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.776742 28709 leader_election.cc:288] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:52.777354 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6227a1f28614fbcb1b5a573d125e670" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:52.777467 27032 raft_consensus.cc:3037] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.778360 27032 raft_consensus.cc:2454] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:52.778607 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6227a1f28614fbcb1b5a573d125e670" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:52.778713 27278 raft_consensus.cc:3037] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.779589 27278 raft_consensus.cc:2454] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:52.780041 24662 leader_election.cc:302] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:52.780303 26677 raft_consensus.cc:2781] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:52.780398 26677 raft_consensus.cc:686] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:52.780519 26677 consensus_queue.cc:237] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:52.782702 24452 catalog_manager.cc:5484] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:52.818565 24595 consensus_queue.cc:237] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:52.820051 24725 raft_consensus.cc:1261] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.820075 24857 raft_consensus.cc:1261] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.820485 26885 consensus_queue.cc:1035] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.820686 29973 consensus_queue.cc:1035] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.822580 29973 raft_consensus.cc:2932] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
W20240501 19:56:52.823864 24531 consensus_peers.cc:485] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ed4b95e555a46c5976ca9e084737ea1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.824288 24857 raft_consensus.cc:2932] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:52.824532 24725 raft_consensus.cc:2932] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:52.827195 24453 catalog_manager.cc:5484] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:52.845770 24857 consensus_queue.cc:237] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:52.847143 24725 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.847249 24595 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:52.847335 24793 consensus_peers.cc:485] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9acd3c8577694e799d3f67a0f2f6adcc. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.847615 30150 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.847790 29519 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.849618 29519 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:52.850008 24595 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:52.851327 24725 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:52.852831 24454 catalog_manager.cc:5484] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:52.855919 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:52.861668 30249 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:52.862560 29972 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:52.864352 30250 tablet.cc:1722] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:52.868000 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=d50b80e5ee3649468d515f2d0b4a0344] alter complete (version 3)
I20240501 19:56:52.877745 24725 consensus_queue.cc:237] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:52.879221 24595 raft_consensus.cc:1261] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.879251 24857 raft_consensus.cc:1261] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:52.879257 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e75026e7e70548e190f81bf79fb30926. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.879587 28709 consensus_queue.cc:1035] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.879820 30357 consensus_queue.cc:1035] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.881454 30357 raft_consensus.cc:2932] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:52.881811 24857 raft_consensus.cc:2932] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:52.883628 24595 raft_consensus.cc:2932] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:52.885368 24452 catalog_manager.cc:5484] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:52.904356 24725 consensus_queue.cc:237] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:52.905850 27032 raft_consensus.cc:1261] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:52.905968 24857 raft_consensus.cc:1261] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:52.906251 24660 consensus_peers.cc:485] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a6b9bbaf2a0470482f3c5ff98322b90. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.906286 26677 consensus_queue.cc:1035] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:52.906430 30357 consensus_queue.cc:1035] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:52.908140 30357 raft_consensus.cc:2932] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:52.908613 27032 raft_consensus.cc:2932] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:52.910212 24857 raft_consensus.cc:2932] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:52.911371 24453 catalog_manager.cc:5484] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:52.970345 24453 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:52.970520 24453 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:52.972817 24453 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet c783f332669845cf93df04cfed4bee0f
I20240501 19:56:52.973305 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet c783f332669845cf93df04cfed4bee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:52 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:52.973502 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet c783f332669845cf93df04cfed4bee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:52 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:52.973637 30415 tablet_replica.cc:330] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:52.973637 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet c783f332669845cf93df04cfed4bee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:52 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:52.973875 30415 raft_consensus.cc:2229] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:52.973989 30417 tablet_replica.cc:330] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:52.974133 30415 raft_consensus.cc:2258] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.974186 30417 raft_consensus.cc:2229] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.974383 30417 raft_consensus.cc:2258] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.974733 30416 tablet_replica.cc:330] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:52.974912 30416 raft_consensus.cc:2229] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.975193 30416 raft_consensus.cc:2258] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.977211 30416 ts_tablet_manager.cc:1875] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:52.977391 30417 ts_tablet_manager.cc:1875] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:52.978364 30415 ts_tablet_manager.cc:1875] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:52.980760 30415 ts_tablet_manager.cc:1888] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:52.980922 30415 log.cc:1196] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/c783f332669845cf93df04cfed4bee0f
I20240501 19:56:52.981285 30415 ts_tablet_manager.cc:1909] T c783f332669845cf93df04cfed4bee0f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:52.982092 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet c783f332669845cf93df04cfed4bee0f (table rebalancer_dynamic_table_new_name [id=d50b80e5ee3649468d515f2d0b4a0344]) successfully deleted
I20240501 19:56:52.983393 30417 ts_tablet_manager.cc:1888] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:52.983471 30417 log.cc:1196] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/c783f332669845cf93df04cfed4bee0f
I20240501 19:56:52.983793 30417 ts_tablet_manager.cc:1909] T c783f332669845cf93df04cfed4bee0f P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:52.984522 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet c783f332669845cf93df04cfed4bee0f (table rebalancer_dynamic_table_new_name [id=d50b80e5ee3649468d515f2d0b4a0344]) successfully deleted
I20240501 19:56:52.986696 30416 ts_tablet_manager.cc:1888] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:52.986832 30416 log.cc:1196] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/c783f332669845cf93df04cfed4bee0f
I20240501 19:56:52.987210 30416 ts_tablet_manager.cc:1909] T c783f332669845cf93df04cfed4bee0f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:52.987985 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet c783f332669845cf93df04cfed4bee0f (table rebalancer_dynamic_table_new_name [id=d50b80e5ee3649468d515f2d0b4a0344]) successfully deleted
I20240501 19:56:53.038528 30085 raft_consensus.cc:482] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:53.038657 30085 raft_consensus.cc:504] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:53.039023 30085 leader_election.cc:288] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:53.039486 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d30871b80a4baa8c10489b752b0b63" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:53.039507 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d30871b80a4baa8c10489b752b0b63" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:53.039642 24857 raft_consensus.cc:2454] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:53.039641 27162 raft_consensus.cc:2454] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:53.039934 27228 leader_election.cc:302] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:53.040083 30085 raft_consensus.cc:2781] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:53.040143 30085 raft_consensus.cc:482] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:53.040182 30085 raft_consensus.cc:3037] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.041168 30085 raft_consensus.cc:504] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:53.041493 30085 leader_election.cc:288] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:53.041767 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d30871b80a4baa8c10489b752b0b63" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:53.041842 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d30871b80a4baa8c10489b752b0b63" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:53.041875 24857 raft_consensus.cc:3037] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.041956 27162 raft_consensus.cc:3037] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.042676 27162 raft_consensus.cc:2454] T 30d30871b80a4baa8c10489b752b0b63 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:53.042923 27229 leader_election.cc:302] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:53.043016 24857 raft_consensus.cc:2454] T 30d30871b80a4baa8c10489b752b0b63 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:53.043090 30085 raft_consensus.cc:2781] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:53.043223 30085 raft_consensus.cc:686] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:53.043361 30085 consensus_queue.cc:237] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:53.045274 24452 catalog_manager.cc:5484] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:53.067911 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:53.071019 27405 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:53.075321 27274 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.077054 24452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:53.081032 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:53.085711 27142 tablet_service.cc:1467] Processing CreateTablet for tablet acf9e03dee7342bc92d1316be95f6075 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13]), partition=
I20240501 19:56:53.085768 27405 tablet_service.cc:1467] Processing CreateTablet for tablet acf9e03dee7342bc92d1316be95f6075 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13]), partition=
I20240501 19:56:53.085970 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf9e03dee7342bc92d1316be95f6075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.085984 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf9e03dee7342bc92d1316be95f6075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.087736 30224 tablet_bootstrap.cc:492] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:53.087914 27274 tablet_service.cc:1467] Processing CreateTablet for tablet acf9e03dee7342bc92d1316be95f6075 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13]), partition=
I20240501 19:56:53.088120 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf9e03dee7342bc92d1316be95f6075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.088646 30224 tablet_bootstrap.cc:654] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.090524 30221 tablet_bootstrap.cc:492] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:53.091373 30221 tablet_bootstrap.cc:654] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.092577 30221 tablet_bootstrap.cc:492] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:53.092645 30221 ts_tablet_manager.cc:1367] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:53.093050 30221 raft_consensus.cc:348] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.093143 30221 raft_consensus.cc:374] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.093170 30221 raft_consensus.cc:729] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.093276 30221 consensus_queue.cc:260] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.093518 30221 ts_tablet_manager.cc:1398] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.093951 30430 tablet_bootstrap.cc:492] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:53.094815 30430 tablet_bootstrap.cc:654] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.097236 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:53.097837 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:53.099778 24966 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.101830 30430 tablet_bootstrap.cc:492] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:53.101961 30430 ts_tablet_manager.cc:1367] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:53.102541 30430 raft_consensus.cc:348] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.102664 30430 raft_consensus.cc:374] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.102691 30430 raft_consensus.cc:729] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.102843 30430 consensus_queue.cc:260] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.103206 30430 ts_tablet_manager.cc:1398] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.127579 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.131708 30224 tablet_bootstrap.cc:492] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:53.131829 30224 ts_tablet_manager.cc:1367] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240501 19:56:53.132287 30224 raft_consensus.cc:348] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.132390 30224 raft_consensus.cc:374] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.132416 30224 raft_consensus.cc:729] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.132531 30224 consensus_queue.cc:260] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.132791 30224 ts_tablet_manager.cc:1398] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.144990 30378 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.148770 30378 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.197460 27278 consensus_queue.cc:237] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:53.198906 24856 raft_consensus.cc:1261] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:53.199074 27227 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 536a3171eab9403eab069fa86c70a8b5. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.199177 27425 raft_consensus.cc:1261] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:53.199479 30085 consensus_queue.cc:1035] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:53.199648 30272 consensus_queue.cc:1035] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:53.201612 30085 raft_consensus.cc:2932] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:53.201994 27425 raft_consensus.cc:2932] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:53.204536 24446 catalog_manager.cc:5484] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:53.205494 24856 raft_consensus.cc:2932] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:53.223284 24986 consensus_queue.cc:237] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:53.224740 27425 raft_consensus.cc:1261] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:53.224817 27162 raft_consensus.cc:1261] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:53.225170 30474 consensus_queue.cc:1035] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:53.225343 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 166bb900935040bd9f7723e822780e6c. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.225533 30474 consensus_queue.cc:1035] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:53.227211 30473 raft_consensus.cc:2932] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:53.227684 27162 raft_consensus.cc:2932] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:53.228386 30477 ts_tablet_manager.cc:902] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:56:53.229259 30477 tablet_copy_client.cc:308] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:56:53.229739 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 9acd3c8577694e799d3f67a0f2f6adcc from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:44335)
I20240501 19:56:53.229846 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 9acd3c8577694e799d3f67a0f2f6adcc from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:44335: session id = 53e34bb09d8e41129a5683079ff63610-9acd3c8577694e799d3f67a0f2f6adcc
I20240501 19:56:53.230885 24878 tablet_copy_source_session.cc:205] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.231995 30477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acd3c8577694e799d3f67a0f2f6adcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.231989 24452 catalog_manager.cc:5484] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:53.232084 27425 raft_consensus.cc:2932] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:53.235272 30477 tablet_copy_client.cc:783] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.235477 30477 tablet_copy_client.cc:647] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.237452 30477 tablet_copy_client.cc:523] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.239101 30477 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:53.259666 24725 consensus_queue.cc:237] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:53.261233 24595 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:53.261551 24856 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:53.261951 26677 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:53.262143 24660 consensus_peers.cc:485] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 224ae792734e4d1ab3c28e20d82fdf14. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.262316 26677 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:53.263991 30378 raft_consensus.cc:2932] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:53.264370 24595 raft_consensus.cc:2932] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:53.264714 30477 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.264977 30477 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:53.265146 30477 ts_tablet_manager.cc:1367] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.003s
I20240501 19:56:53.266369 30477 raft_consensus.cc:348] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:53.266517 30477 raft_consensus.cc:729] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.266674 30477 consensus_queue.cc:260] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:53.267066 30477 ts_tablet_manager.cc:1398] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:53.267493 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-9acd3c8577694e799d3f67a0f2f6adcc received from {username='slave'} at 127.17.65.7:44335
I20240501 19:56:53.267570 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-9acd3c8577694e799d3f67a0f2f6adcc on tablet 9acd3c8577694e799d3f67a0f2f6adcc with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:53.267877 24446 catalog_manager.cc:5484] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:53.268751 24856 raft_consensus.cc:2932] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:53.286823 24725 consensus_queue.cc:237] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:53.288236 27278 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:56:53.288487 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b240b461b548a087fcabdeb938df1e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.288688 28709 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.289319 24595 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.289752 28709 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.291364 26677 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:53.292115 24595 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:53.293888 27278 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:53.294669 24452 catalog_manager.cc:5484] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:53.310180 26677 consensus_queue.cc:1035] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.314149 28709 consensus_queue.cc:1035] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.315955 27162 consensus_queue.cc:237] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:53.317107 30477 ts_tablet_manager.cc:902] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:53.317605 30477 tablet_copy_client.cc:308] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:53.319026 27425 raft_consensus.cc:1261] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:53.319413 30540 consensus_queue.cc:1035] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:53.319736 24986 raft_consensus.cc:1261] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:53.320008 27097 consensus_peers.cc:485] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afc479e18b5e4efa8a8fa113db3dbf1b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.320289 30540 consensus_queue.cc:1035] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:53.321839 30540 raft_consensus.cc:2932] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:53.322230 24986 raft_consensus.cc:2932] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:53.323371 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 0ed4b95e555a46c5976ca9e084737ea1 from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:36057)
I20240501 19:56:53.323474 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 0ed4b95e555a46c5976ca9e084737ea1 from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:36057: session id = 53e34bb09d8e41129a5683079ff63610-0ed4b95e555a46c5976ca9e084737ea1
I20240501 19:56:53.324084 27425 raft_consensus.cc:2932] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:53.324522 24616 tablet_copy_source_session.cc:205] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.325861 24446 catalog_manager.cc:5484] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:53.327324 30477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed4b95e555a46c5976ca9e084737ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.328265 30554 ts_tablet_manager.cc:902] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:53.328815 30554 tablet_copy_client.cc:308] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:53.329485 30477 tablet_copy_client.cc:783] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.329488 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet e75026e7e70548e190f81bf79fb30926 from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:35875)
I20240501 19:56:53.329576 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet e75026e7e70548e190f81bf79fb30926 from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:35875: session id = 29d1a658e2b946e0b412e5fef82db9d9-e75026e7e70548e190f81bf79fb30926
I20240501 19:56:53.329658 30477 tablet_copy_client.cc:647] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.330636 24746 tablet_copy_source_session.cc:205] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.332868 30554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75026e7e70548e190f81bf79fb30926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.335937 30477 tablet_copy_client.cc:523] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.337285 30477 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:53.337706 30554 tablet_copy_client.cc:783] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.337860 30554 tablet_copy_client.cc:647] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.338999 30554 tablet_copy_client.cc:523] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.340209 30554 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:53.353188 24595 consensus_queue.cc:237] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.354710 27031 raft_consensus.cc:1261] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:53.354801 27162 raft_consensus.cc:1261] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:53.354962 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caf81a72d0304b8aa611d4c1eab5ac26. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.355345 29973 consensus_queue.cc:1035] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:53.355677 29973 consensus_queue.cc:1035] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:53.363885 30477 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.364164 30477 tablet_bootstrap.cc:492] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:53.364341 30477 ts_tablet_manager.cc:1367] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20240501 19:56:53.364796 30477 raft_consensus.cc:348] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:53.364912 30554 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.364913 30477 raft_consensus.cc:729] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.365044 30477 consensus_queue.cc:260] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:53.365162 30554 tablet_bootstrap.cc:492] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:56:53.365325 30554 ts_tablet_manager.cc:1367] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.002s
I20240501 19:56:53.365321 30477 ts_tablet_manager.cc:1398] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.365803 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-0ed4b95e555a46c5976ca9e084737ea1 received from {username='slave'} at 127.17.65.7:36057
I20240501 19:56:53.365865 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-0ed4b95e555a46c5976ca9e084737ea1 on tablet 0ed4b95e555a46c5976ca9e084737ea1 with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:53.365792 30554 raft_consensus.cc:348] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:53.366241 30554 raft_consensus.cc:729] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.366402 30554 consensus_queue.cc:260] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:53.366783 30572 raft_consensus.cc:2932] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:53.367588 27031 raft_consensus.cc:2932] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:53.370543 27162 raft_consensus.cc:2932] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:53.370558 24446 catalog_manager.cc:5484] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:53.373277 30554 ts_tablet_manager.cc:1398] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:53.379452 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-e75026e7e70548e190f81bf79fb30926 received from {username='slave'} at 127.17.65.6:35875
I20240501 19:56:53.379530 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-e75026e7e70548e190f81bf79fb30926 on tablet e75026e7e70548e190f81bf79fb30926 with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:56:53.395411 30169 raft_consensus.cc:482] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:53.395599 30169 raft_consensus.cc:504] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.395942 30169 leader_election.cc:288] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:53.396642 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf9e03dee7342bc92d1316be95f6075" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:53.396766 27278 raft_consensus.cc:2454] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:53.397395 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf9e03dee7342bc92d1316be95f6075" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:53.397521 27162 raft_consensus.cc:2454] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:53.397861 27361 leader_election.cc:302] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:53.398284 30169 raft_consensus.cc:2781] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:53.398340 30169 raft_consensus.cc:482] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:53.398382 30169 raft_consensus.cc:3037] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.399493 30169 raft_consensus.cc:504] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.399792 30169 leader_election.cc:288] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:53.400116 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf9e03dee7342bc92d1316be95f6075" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:53.400223 27278 raft_consensus.cc:3037] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.400205 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf9e03dee7342bc92d1316be95f6075" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:53.400296 27162 raft_consensus.cc:3037] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.401098 27278 raft_consensus.cc:2454] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:53.401098 27162 raft_consensus.cc:2454] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:53.401361 27361 leader_election.cc:302] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:53.401618 30169 raft_consensus.cc:2781] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:53.401759 30169 raft_consensus.cc:686] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:53.401891 30169 consensus_queue.cc:237] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.403553 24453 catalog_manager.cc:5484] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:53.462592 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:53.464741 27405 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:53.464761 27142 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:53.470005 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:53.472833 27274 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.480988 24966 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.494261 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:53.500448 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.525553 30630 ts_tablet_manager.cc:902] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:53.526319 30630 tablet_copy_client.cc:308] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:53.526954 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet 9a6b9bbaf2a0470482f3c5ff98322b90 from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:43771)
I20240501 19:56:53.527091 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet 9a6b9bbaf2a0470482f3c5ff98322b90 from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:43771: session id = bbf7735f82ca4f65a5170ac7909faa47-9a6b9bbaf2a0470482f3c5ff98322b90
I20240501 19:56:53.528009 24746 tablet_copy_source_session.cc:205] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.529326 30630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6b9bbaf2a0470482f3c5ff98322b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.531934 30630 tablet_copy_client.cc:783] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.532102 30630 tablet_copy_client.cc:647] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.534560 30630 tablet_copy_client.cc:523] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.536389 30630 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:53.541100 30630 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.541348 30630 tablet_bootstrap.cc:492] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:56:53.541488 30630 ts_tablet_manager.cc:1367] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 19:56:53.541877 30630 raft_consensus.cc:348] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:53.541989 30630 raft_consensus.cc:729] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.542140 30630 consensus_queue.cc:260] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:53.542411 30630 ts_tablet_manager.cc:1398] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:53.543215 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-9a6b9bbaf2a0470482f3c5ff98322b90 received from {username='slave'} at 127.17.65.1:43771
I20240501 19:56:53.543282 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-9a6b9bbaf2a0470482f3c5ff98322b90 on tablet 9a6b9bbaf2a0470482f3c5ff98322b90 with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:56:53.554172 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:53.560461 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 8080df897fa94d199a2e9917a734b8d7 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.560743 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8080df897fa94d199a2e9917a734b8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.562620 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 8080df897fa94d199a2e9917a734b8d7 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.562839 30657 tablet_bootstrap.cc:492] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:53.562894 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8080df897fa94d199a2e9917a734b8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.563261 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 4b133f0ed2014a0f99812e4cd8d0e422 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.563266 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 0366ec1cbd8b4052afc919ac1c452810 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.563519 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b133f0ed2014a0f99812e4cd8d0e422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.563550 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0366ec1cbd8b4052afc919ac1c452810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.565228 30658 tablet_bootstrap.cc:492] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:53.565354 30221 tablet_bootstrap.cc:492] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:53.565582 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 0366ec1cbd8b4052afc919ac1c452810 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.565801 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0366ec1cbd8b4052afc919ac1c452810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.565930 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:53.565941 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 4b133f0ed2014a0f99812e4cd8d0e422 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.566205 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b133f0ed2014a0f99812e4cd8d0e422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.566259 30221 tablet_bootstrap.cc:654] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.568186 30657 tablet_bootstrap.cc:654] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.568565 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 4b133f0ed2014a0f99812e4cd8d0e422 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.568862 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b133f0ed2014a0f99812e4cd8d0e422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.569187 30657 tablet_bootstrap.cc:492] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:53.569260 30657 ts_tablet_manager.cc:1367] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:53.569679 30657 raft_consensus.cc:348] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.569788 30658 tablet_bootstrap.cc:654] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.569784 30657 raft_consensus.cc:374] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.569818 30657 raft_consensus.cc:729] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.569927 30657 consensus_queue.cc:260] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.570223 30657 ts_tablet_manager.cc:1398] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.570374 30657 tablet_bootstrap.cc:492] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:53.571027 30430 tablet_bootstrap.cc:492] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:53.571112 30658 tablet_bootstrap.cc:492] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:53.571184 30658 ts_tablet_manager.cc:1367] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:53.571262 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 0366ec1cbd8b4052afc919ac1c452810 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.571506 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0366ec1cbd8b4052afc919ac1c452810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.571565 30658 raft_consensus.cc:348] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.571662 30658 raft_consensus.cc:374] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.571705 30658 raft_consensus.cc:729] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.571825 30658 consensus_queue.cc:260] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.571920 30430 tablet_bootstrap.cc:654] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.572136 30658 ts_tablet_manager.cc:1398] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.573288 30657 tablet_bootstrap.cc:654] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.573920 30224 tablet_bootstrap.cc:492] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:53.574846 30224 tablet_bootstrap.cc:654] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.575724 30657 tablet_bootstrap.cc:492] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:53.575794 30657 ts_tablet_manager.cc:1367] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:53.576228 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 8080df897fa94d199a2e9917a734b8d7 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=79b39cc0b0e043519d550a38029fa9d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:53.576221 30657 raft_consensus.cc:348] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.576314 30657 raft_consensus.cc:374] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.576354 30657 raft_consensus.cc:729] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.576426 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8080df897fa94d199a2e9917a734b8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.576468 30657 consensus_queue.cc:260] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.576757 30657 ts_tablet_manager.cc:1398] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.579875 30659 tablet_bootstrap.cc:492] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:53.580866 30659 tablet_bootstrap.cc:654] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.582126 30430 tablet_bootstrap.cc:492] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:53.582201 30430 ts_tablet_manager.cc:1367] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:53.582454 30221 tablet_bootstrap.cc:492] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:53.582544 30221 ts_tablet_manager.cc:1367] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:56:53.582571 30430 raft_consensus.cc:348] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.582679 30430 raft_consensus.cc:374] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.582705 30430 raft_consensus.cc:729] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.582798 30430 consensus_queue.cc:260] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.583020 30430 ts_tablet_manager.cc:1398] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.583177 30430 tablet_bootstrap.cc:492] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:53.584014 30430 tablet_bootstrap.cc:654] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.583019 30221 raft_consensus.cc:348] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.584214 30221 raft_consensus.cc:374] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.584254 30221 raft_consensus.cc:729] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.584367 30221 consensus_queue.cc:260] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.584664 30221 ts_tablet_manager.cc:1398] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:53.583890 30272 consensus_queue.cc:1035] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 19:56:53.590400 29464 consensus_queue.cc:1035] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.603827 28709 raft_consensus.cc:482] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:53.603917 28709 raft_consensus.cc:504] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.604152 27276 raft_consensus.cc:1261] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:53.604256 27163 raft_consensus.cc:1261] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:53.604328 28709 leader_election.cc:288] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:53.604738 30587 consensus_queue.cc:1035] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.604926 30659 tablet_bootstrap.cc:492] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:53.605022 30659 ts_tablet_manager.cc:1367] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 19:56:53.605017 30587 consensus_queue.cc:1035] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.605331 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8080df897fa94d199a2e9917a734b8d7" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:53.605468 24595 raft_consensus.cc:2454] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:53.605473 30659 raft_consensus.cc:348] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.605839 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8080df897fa94d199a2e9917a734b8d7" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:53.606194 24660 leader_election.cc:302] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:53.607326 28709 raft_consensus.cc:2781] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:53.607381 28709 raft_consensus.cc:482] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:53.607422 28709 raft_consensus.cc:3037] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.607477 24449 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet acf9e03dee7342bc92d1316be95f6075 (table rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13]). Expected version 1 got 0
I20240501 19:56:53.605573 30659 raft_consensus.cc:374] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.608415 30659 raft_consensus.cc:729] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.608422 28709 raft_consensus.cc:504] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.608559 30659 consensus_queue.cc:260] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.608727 28709 leader_election.cc:288] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:53.608866 30659 ts_tablet_manager.cc:1398] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:53.609004 30659 tablet_bootstrap.cc:492] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:53.609612 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8080df897fa94d199a2e9917a734b8d7" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:53.609725 24595 raft_consensus.cc:3037] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.610774 24595 raft_consensus.cc:2454] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:53.612239 24660 leader_election.cc:302] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:53.612576 26677 raft_consensus.cc:2781] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:53.612660 26677 raft_consensus.cc:686] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:53.612782 26677 consensus_queue.cc:237] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.613716 30659 tablet_bootstrap.cc:654] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:53.614648 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8080df897fa94d199a2e9917a734b8d7" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
W20240501 19:56:53.614935 24662 leader_election.cc:341] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:56:53.615480 24662 leader_election.cc:341] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:53.617295 30430 tablet_bootstrap.cc:492] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:53.617391 30430 ts_tablet_manager.cc:1367] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.001s
I20240501 19:56:53.617831 30430 raft_consensus.cc:348] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.617925 30430 raft_consensus.cc:374] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.617964 30430 raft_consensus.cc:729] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.618113 24446 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet acf9e03dee7342bc92d1316be95f6075 (table rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13]). Expected version 1 got 0
I20240501 19:56:53.618106 30430 consensus_queue.cc:260] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.619323 30224 tablet_bootstrap.cc:492] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:53.619454 30224 ts_tablet_manager.cc:1367] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.001s
I20240501 19:56:53.619695 30430 ts_tablet_manager.cc:1398] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:53.614996 24445 catalog_manager.cc:5484] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:53.619910 30224 raft_consensus.cc:348] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.620005 30224 raft_consensus.cc:374] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.620038 30224 raft_consensus.cc:729] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.620144 30224 consensus_queue.cc:260] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.620395 30224 ts_tablet_manager.cc:1398] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:53.627135 24447 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet acf9e03dee7342bc92d1316be95f6075 (table rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13]). Expected version 1 got 0
I20240501 19:56:53.629796 30696 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:53.630353 30697 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:53.631717 30696 tablet.cc:1620] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:53.632195 30697 tablet.cc:1620] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:53.637893 30659 tablet_bootstrap.cc:492] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:53.638005 30659 ts_tablet_manager.cc:1367] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 19:56:53.638525 30659 raft_consensus.cc:348] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.638670 30659 raft_consensus.cc:374] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:53.638713 30659 raft_consensus.cc:729] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.638849 30659 consensus_queue.cc:260] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.639186 30659 ts_tablet_manager.cc:1398] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:53.642320 24449 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet acf9e03dee7342bc92d1316be95f6075 (table rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13]). Expected version 1 got 0
I20240501 19:56:53.643685 30698 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:53.643997 30698 tablet.cc:1620] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:53.647498 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13] alter complete (version 1)
I20240501 19:56:53.704653 30699 ts_tablet_manager.cc:902] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:53.705487 30699 tablet_copy_client.cc:308] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.6:44579
I20240501 19:56:53.706184 27183 tablet_copy_service.cc:140] P 29d1a658e2b946e0b412e5fef82db9d9: Received BeginTabletCopySession request for tablet afc479e18b5e4efa8a8fa113db3dbf1b from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:50333)
I20240501 19:56:53.706301 27183 tablet_copy_service.cc:161] P 29d1a658e2b946e0b412e5fef82db9d9: Beginning new tablet copy session on tablet afc479e18b5e4efa8a8fa113db3dbf1b from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:50333: session id = 9d64cdfaeb2d46b29845160e395fc241-afc479e18b5e4efa8a8fa113db3dbf1b
I20240501 19:56:53.707531 27183 tablet_copy_source_session.cc:205] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.708894 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc479e18b5e4efa8a8fa113db3dbf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.711107 30699 tablet_copy_client.cc:783] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.711323 30699 tablet_copy_client.cc:647] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.712787 30699 tablet_copy_client.cc:523] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.714088 30699 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:53.717525 30699 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.717792 30699 tablet_bootstrap.cc:492] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:56:53.717944 30699 ts_tablet_manager.cc:1367] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:53.718333 30699 raft_consensus.cc:348] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:53.718430 30699 raft_consensus.cc:729] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.718546 30699 consensus_queue.cc:260] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:53.718814 30699 ts_tablet_manager.cc:1398] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.719403 27183 tablet_copy_service.cc:342] P 29d1a658e2b946e0b412e5fef82db9d9: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-afc479e18b5e4efa8a8fa113db3dbf1b received from {username='slave'} at 127.17.65.3:50333
I20240501 19:56:53.719499 27183 tablet_copy_service.cc:433] P 29d1a658e2b946e0b412e5fef82db9d9: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-afc479e18b5e4efa8a8fa113db3dbf1b on tablet afc479e18b5e4efa8a8fa113db3dbf1b with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:56:53.746708 30554 ts_tablet_manager.cc:902] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:53.747296 30554 tablet_copy_client.cc:308] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:53.747792 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet a4b240b461b548a087fcabdeb938df1e from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:35875)
I20240501 19:56:53.747896 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet a4b240b461b548a087fcabdeb938df1e from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:35875: session id = 29d1a658e2b946e0b412e5fef82db9d9-a4b240b461b548a087fcabdeb938df1e
I20240501 19:56:53.748849 24746 tablet_copy_source_session.cc:205] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.749706 30554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b240b461b548a087fcabdeb938df1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.751837 30554 tablet_copy_client.cc:783] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.751986 30554 tablet_copy_client.cc:647] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.753123 30554 tablet_copy_client.cc:523] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.754230 30554 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:53.756525 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:53.765404 30696 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:53.765717 30696 tablet.cc:1620] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:53.766741 30698 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:53.767009 30698 tablet.cc:1620] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:53.772682 30697 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:53.772986 30697 tablet.cc:1620] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:53.776800 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=c4553772ef1d433bbbcdb79923f00d13] alter complete (version 2)
I20240501 19:56:53.785010 30481 raft_consensus.cc:482] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:53.785105 30481 raft_consensus.cc:504] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.785449 30481 leader_election.cc:288] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:53.785750 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0366ec1cbd8b4052afc919ac1c452810" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:53.785863 27276 raft_consensus.cc:2454] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:53.785964 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0366ec1cbd8b4052afc919ac1c452810" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:53.786111 24595 raft_consensus.cc:2454] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:53.786410 24924 leader_election.cc:302] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:53.786724 30481 raft_consensus.cc:2781] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:53.786787 30481 raft_consensus.cc:482] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:53.786834 30481 raft_consensus.cc:3037] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.787931 30481 raft_consensus.cc:504] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.788260 30481 leader_election.cc:288] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:53.788563 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0366ec1cbd8b4052afc919ac1c452810" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:53.788580 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0366ec1cbd8b4052afc919ac1c452810" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:53.788671 24595 raft_consensus.cc:3037] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.788715 27276 raft_consensus.cc:3037] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.789636 24595 raft_consensus.cc:2454] T 0366ec1cbd8b4052afc919ac1c452810 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:53.789671 27276 raft_consensus.cc:2454] T 0366ec1cbd8b4052afc919ac1c452810 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:53.790066 24922 leader_election.cc:302] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:53.790442 30481 raft_consensus.cc:2781] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:53.790618 30481 raft_consensus.cc:686] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:53.790748 30481 consensus_queue.cc:237] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:53.792647 24445 catalog_manager.cc:5484] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:53.793953 30712 ts_tablet_manager.cc:902] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:53.794492 30712 tablet_copy_client.cc:308] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:56:53.794649 30554 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.794939 30554 tablet_bootstrap.cc:492] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:56:53.795143 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet 536a3171eab9403eab069fa86c70a8b5 from peer 9081292597be4a72a42b216fddcd7971 ({username='slave'} at 127.17.65.2:41119)
I20240501 19:56:53.795182 30554 ts_tablet_manager.cc:1367] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.003s
I20240501 19:56:53.795233 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet 536a3171eab9403eab069fa86c70a8b5 from peer 9081292597be4a72a42b216fddcd7971 at {username='slave'} at 127.17.65.2:41119: session id = 9081292597be4a72a42b216fddcd7971-536a3171eab9403eab069fa86c70a8b5
I20240501 19:56:53.795624 30554 raft_consensus.cc:348] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:53.795732 30554 raft_consensus.cc:729] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.795792 27276 raft_consensus.cc:1203] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:53.795883 30554 consensus_queue.cc:260] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:53.796232 30554 ts_tablet_manager.cc:1398] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.796545 27314 tablet_copy_source_session.cc:205] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.797462 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-a4b240b461b548a087fcabdeb938df1e received from {username='slave'} at 127.17.65.6:35875
I20240501 19:56:53.797529 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-a4b240b461b548a087fcabdeb938df1e on tablet a4b240b461b548a087fcabdeb938df1e with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:56:53.799584 30712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536a3171eab9403eab069fa86c70a8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.801656 30712 tablet_copy_client.cc:783] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.801991 30712 tablet_copy_client.cc:647] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.802984 30712 tablet_copy_client.cc:523] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.803949 30712 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:53.807009 30712 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.807262 30712 tablet_bootstrap.cc:492] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: Bootstrap complete.
I20240501 19:56:53.807404 30712 ts_tablet_manager.cc:1367] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:53.807770 30712 raft_consensus.cc:348] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:53.807860 30712 raft_consensus.cc:729] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.807976 30712 consensus_queue.cc:260] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:53.808228 30712 ts_tablet_manager.cc:1398] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.808637 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session 9081292597be4a72a42b216fddcd7971-536a3171eab9403eab069fa86c70a8b5 received from {username='slave'} at 127.17.65.2:41119
I20240501 19:56:53.808727 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session 9081292597be4a72a42b216fddcd7971-536a3171eab9403eab069fa86c70a8b5 on tablet 536a3171eab9403eab069fa86c70a8b5 with peer 9081292597be4a72a42b216fddcd7971
I20240501 19:56:53.827232 30541 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:53.827376 30541 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.827751 30541 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:53.828089 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b133f0ed2014a0f99812e4cd8d0e422" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:53.828238 24987 raft_consensus.cc:2454] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:53.828397 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b133f0ed2014a0f99812e4cd8d0e422" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:53.828536 27425 raft_consensus.cc:2454] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:53.828594 27097 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:53.828763 30541 raft_consensus.cc:2781] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:53.828827 30541 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:53.828876 30541 raft_consensus.cc:3037] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.830060 30541 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.830459 30541 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:53.830780 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b133f0ed2014a0f99812e4cd8d0e422" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:53.830822 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b133f0ed2014a0f99812e4cd8d0e422" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:53.830901 27425 raft_consensus.cc:3037] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.830919 24987 raft_consensus.cc:3037] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.831920 24987 raft_consensus.cc:2454] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:53.831920 27425 raft_consensus.cc:2454] T 4b133f0ed2014a0f99812e4cd8d0e422 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:53.832201 27097 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:53.832351 30541 raft_consensus.cc:2781] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:53.832515 30541 raft_consensus.cc:686] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:53.832665 30541 consensus_queue.cc:237] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.833784 30477 ts_tablet_manager.cc:902] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:53.834250 30477 tablet_copy_client.cc:308] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:56:53.834781 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet 166bb900935040bd9f7723e822780e6c from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:37883)
I20240501 19:56:53.834877 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet 166bb900935040bd9f7723e822780e6c from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:37883: session id = 53e34bb09d8e41129a5683079ff63610-166bb900935040bd9f7723e822780e6c
I20240501 19:56:53.835289 24447 catalog_manager.cc:5484] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:53.835772 25009 tablet_copy_source_session.cc:205] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.837239 30477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166bb900935040bd9f7723e822780e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.839398 30477 tablet_copy_client.cc:783] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.839522 30587 raft_consensus.cc:482] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:53.839565 30477 tablet_copy_client.cc:647] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.839597 30587 raft_consensus.cc:504] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.839963 30587 leader_election.cc:288] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:53.840281 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8080df897fa94d199a2e9917a734b8d7" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:53.840286 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8080df897fa94d199a2e9917a734b8d7" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:53.840427 24595 raft_consensus.cc:2379] T 8080df897fa94d199a2e9917a734b8d7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a6e54e86a9949e5970b42e8e32e7561 in current term 1: Already voted for candidate 9081292597be4a72a42b216fddcd7971 in this term.
I20240501 19:56:53.840962 27359 leader_election.cc:302] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561; no voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:56:53.841038 30477 tablet_copy_client.cc:523] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.841177 30587 raft_consensus.cc:3037] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.841948 30587 raft_consensus.cc:2726] T 8080df897fa94d199a2e9917a734b8d7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:53.842090 30477 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:53.845005 30477 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.845211 30477 tablet_bootstrap.cc:492] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:56:53.845345 30477 ts_tablet_manager.cc:1367] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:53.845697 30477 raft_consensus.cc:348] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:53.845819 30477 raft_consensus.cc:729] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.845952 30477 consensus_queue.cc:260] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:53.846324 30477 ts_tablet_manager.cc:1398] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.846810 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-166bb900935040bd9f7723e822780e6c received from {username='slave'} at 127.17.65.7:37883
I20240501 19:56:53.846894 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-166bb900935040bd9f7723e822780e6c on tablet 166bb900935040bd9f7723e822780e6c with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:53.857589 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:53.861006 27142 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.865164 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:53.875660 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.876788 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:53.877816 27406 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:53.884362 24447 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:53.887492 27274 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.891927 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:53.896986 30696 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:53.897207 30698 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:53.898001 30697 tablet.cc:1722] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:53.902966 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=c4553772ef1d433bbbcdb79923f00d13] alter complete (version 3)
I20240501 19:56:53.904072 24595 raft_consensus.cc:1203] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:53.916069 30736 ts_tablet_manager.cc:902] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:53.916793 30736 tablet_copy_client.cc:308] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:53.919463 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet 224ae792734e4d1ab3c28e20d82fdf14 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:37455)
I20240501 19:56:53.919577 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet 224ae792734e4d1ab3c28e20d82fdf14 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:37455: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-224ae792734e4d1ab3c28e20d82fdf14
I20240501 19:56:53.920564 24746 tablet_copy_source_session.cc:205] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.921944 30736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224ae792734e4d1ab3c28e20d82fdf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.922127 27276 raft_consensus.cc:1203] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:53.935956 30736 tablet_copy_client.cc:783] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.936693 30736 tablet_copy_client.cc:647] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.937901 27163 raft_consensus.cc:1203] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:53.938063 30736 tablet_copy_client.cc:523] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.940618 30736 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:53.941819 30741 ts_tablet_manager.cc:902] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:53.942407 30741 tablet_copy_client.cc:308] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:53.942873 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet caf81a72d0304b8aa611d4c1eab5ac26 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:56477)
I20240501 19:56:53.942986 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet caf81a72d0304b8aa611d4c1eab5ac26 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:56477: session id = 4a6e54e86a9949e5970b42e8e32e7561-caf81a72d0304b8aa611d4c1eab5ac26
I20240501 19:56:53.944008 24616 tablet_copy_source_session.cc:205] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.945807 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf81a72d0304b8aa611d4c1eab5ac26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.949955 30741 tablet_copy_client.cc:783] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.950538 30741 tablet_copy_client.cc:647] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.951614 30741 tablet_copy_client.cc:523] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.952822 30741 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:53.957342 30741 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.957557 30741 tablet_bootstrap.cc:492] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:56:53.957713 30741 ts_tablet_manager.cc:1367] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:53.958081 30741 raft_consensus.cc:348] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.958169 30741 raft_consensus.cc:729] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.958274 30741 consensus_queue.cc:260] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:53.958544 30741 ts_tablet_manager.cc:1398] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.959074 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-caf81a72d0304b8aa611d4c1eab5ac26 received from {username='slave'} at 127.17.65.8:56477
I20240501 19:56:53.959151 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-caf81a72d0304b8aa611d4c1eab5ac26 on tablet caf81a72d0304b8aa611d4c1eab5ac26 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:53.967525 30736 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.967784 30736 tablet_bootstrap.cc:492] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:56:53.967957 30736 ts_tablet_manager.cc:1367] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20240501 19:56:53.968367 30736 raft_consensus.cc:348] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:53.968482 30736 raft_consensus.cc:729] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:53.968618 30736 consensus_queue.cc:260] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:53.968916 30736 ts_tablet_manager.cc:1398] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.969420 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-224ae792734e4d1ab3c28e20d82fdf14 received from {username='slave'} at 127.17.65.5:37455
I20240501 19:56:53.969499 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-224ae792734e4d1ab3c28e20d82fdf14 on tablet 224ae792734e4d1ab3c28e20d82fdf14 with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:54.007000 24449 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:54.007161 24449 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:54.010246 24449 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet acf9e03dee7342bc92d1316be95f6075
I20240501 19:56:54.010850 27406 tablet_service.cc:1514] Processing DeleteTablet for tablet acf9e03dee7342bc92d1316be95f6075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:54 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:54.010939 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet acf9e03dee7342bc92d1316be95f6075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:54 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:56:54.011245 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet acf9e03dee7342bc92d1316be95f6075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:54 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:54.011456 30789 tablet_replica.cc:330] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:54.011667 30789 raft_consensus.cc:2229] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:54.011950 30789 raft_consensus.cc:2258] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.012641 30789 ts_tablet_manager.cc:1875] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:54.013860 30791 tablet_replica.cc:330] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:54.014034 30790 tablet_replica.cc:330] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:56:54.014219 30791 raft_consensus.cc:2229] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.014416 30790 raft_consensus.cc:2229] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.014593 30790 raft_consensus.cc:2258] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.014775 30791 raft_consensus.cc:2258] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.015555 30791 ts_tablet_manager.cc:1875] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:54.015568 30790 ts_tablet_manager.cc:1875] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:54.019150 30789 ts_tablet_manager.cc:1888] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:54.019273 30789 log.cc:1196] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/acf9e03dee7342bc92d1316be95f6075
I20240501 19:56:54.021828 30789 ts_tablet_manager.cc:1909] T acf9e03dee7342bc92d1316be95f6075 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:54.024173 30790 ts_tablet_manager.cc:1888] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:54.024288 30790 log.cc:1196] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/acf9e03dee7342bc92d1316be95f6075
I20240501 19:56:54.024616 30790 ts_tablet_manager.cc:1909] T acf9e03dee7342bc92d1316be95f6075 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:56:54.025568 30791 ts_tablet_manager.cc:1888] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:54.025655 30791 log.cc:1196] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/acf9e03dee7342bc92d1316be95f6075
I20240501 19:56:54.025966 30791 ts_tablet_manager.cc:1909] T acf9e03dee7342bc92d1316be95f6075 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:54.025964 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet acf9e03dee7342bc92d1316be95f6075 (table rebalancer_dynamic_table_new_name [id=c4553772ef1d433bbbcdb79923f00d13]) successfully deleted
I20240501 19:56:54.026108 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet acf9e03dee7342bc92d1316be95f6075 (table rebalancer_dynamic_table_new_name [id=c4553772ef1d433bbbcdb79923f00d13]) successfully deleted
I20240501 19:56:54.026614 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet acf9e03dee7342bc92d1316be95f6075 (table rebalancer_dynamic_table_new_name [id=c4553772ef1d433bbbcdb79923f00d13]) successfully deleted
I20240501 19:56:54.046243 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:54.046473 27406 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:54.051658 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:54.062237 27142 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:54.062543 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:54.062809 27274 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.075805 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.083207 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:54.106750 30357 consensus_queue.cc:1035] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:54.110196 30378 consensus_queue.cc:1035] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:54.112921 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:54.118170 24837 tablet_service.cc:1467] Processing CreateTablet for tablet e63440e636484ab481007ef81ceceaa3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2224d9fb54b44cebe21304c7b1df9f5]), partition=
I20240501 19:56:54.118465 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63440e636484ab481007ef81ceceaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.118615 27406 tablet_service.cc:1467] Processing CreateTablet for tablet e63440e636484ab481007ef81ceceaa3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2224d9fb54b44cebe21304c7b1df9f5]), partition=
I20240501 19:56:54.118851 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63440e636484ab481007ef81ceceaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.121034 30814 tablet_bootstrap.cc:492] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:54.121302 27142 tablet_service.cc:1467] Processing CreateTablet for tablet e63440e636484ab481007ef81ceceaa3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2224d9fb54b44cebe21304c7b1df9f5]), partition=
I20240501 19:56:54.121573 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63440e636484ab481007ef81ceceaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.122073 30814 tablet_bootstrap.cc:654] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.122077 30815 tablet_bootstrap.cc:492] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:54.122962 30815 tablet_bootstrap.cc:654] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.124068 30815 tablet_bootstrap.cc:492] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:54.124166 30815 ts_tablet_manager.cc:1367] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:54.124760 30815 raft_consensus.cc:348] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.124969 30815 raft_consensus.cc:374] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.124980 30814 tablet_bootstrap.cc:492] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:54.125073 30814 ts_tablet_manager.cc:1367] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:54.125061 30815 raft_consensus.cc:729] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.125216 30815 consensus_queue.cc:260] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.125528 30814 raft_consensus.cc:348] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.125629 30814 raft_consensus.cc:374] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.125658 30814 raft_consensus.cc:729] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.125766 30814 consensus_queue.cc:260] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.126010 30814 ts_tablet_manager.cc:1398] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:54.126113 30815 ts_tablet_manager.cc:1398] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:54.126516 30818 tablet_bootstrap.cc:492] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:54.128836 30818 tablet_bootstrap.cc:654] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.129798 30818 tablet_bootstrap.cc:492] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:54.129879 30818 ts_tablet_manager.cc:1367] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:54.130324 30818 raft_consensus.cc:348] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.130488 30818 raft_consensus.cc:374] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.130540 30818 raft_consensus.cc:729] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.130892 30818 consensus_queue.cc:260] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.131232 30818 ts_tablet_manager.cc:1398] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:54.197986 24725 raft_consensus.cc:1203] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:54.210474 30541 raft_consensus.cc:482] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:54.210608 30541 raft_consensus.cc:504] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.211052 30541 leader_election.cc:288] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:54.211388 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63440e636484ab481007ef81ceceaa3" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:54.211534 24856 raft_consensus.cc:2454] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:54.211516 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63440e636484ab481007ef81ceceaa3" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:54.211634 27425 raft_consensus.cc:2454] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:54.211818 27097 leader_election.cc:302] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:54.212077 30541 raft_consensus.cc:2781] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:54.212138 30541 raft_consensus.cc:482] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:54.212183 30541 raft_consensus.cc:3037] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.212967 30541 raft_consensus.cc:504] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.213268 30541 leader_election.cc:288] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:54.213536 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63440e636484ab481007ef81ceceaa3" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:54.213593 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63440e636484ab481007ef81ceceaa3" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:54.213644 24856 raft_consensus.cc:3037] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.213733 27425 raft_consensus.cc:3037] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.214401 24856 raft_consensus.cc:2454] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:54.214498 27425 raft_consensus.cc:2454] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:54.214762 27098 leader_election.cc:302] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:54.214905 30541 raft_consensus.cc:2781] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:54.214982 30541 raft_consensus.cc:686] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:54.215138 30541 consensus_queue.cc:237] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.216959 24451 catalog_manager.cc:5484] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:54.221711 24856 raft_consensus.cc:1203] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:54.224751 30481 consensus_queue.cc:1035] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:54.230154 30710 consensus_queue.cc:1035] T 0366ec1cbd8b4052afc919ac1c452810 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:54.239795 27142 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.243712 27012 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:54.245349 27274 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.246310 27163 raft_consensus.cc:1203] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:56:54.248644 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.256819 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:54.260900 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:54.262413 27406 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:54.272467 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:54.291894 27276 raft_consensus.cc:1203] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:54.303234 30541 consensus_queue.cc:1035] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:54.307933 30718 consensus_queue.cc:1035] T 4b133f0ed2014a0f99812e4cd8d0e422 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:54.325508 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:54.329469 27425 raft_consensus.cc:1261] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:54.329519 24856 raft_consensus.cc:1261] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:54.329926 30718 consensus_queue.cc:1035] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:54.330060 30541 consensus_queue.cc:1035] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:54.332890 24450 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet e63440e636484ab481007ef81ceceaa3 (table rebalancer_dynamic_table [id=d2224d9fb54b44cebe21304c7b1df9f5]). Expected version 1 got 0
I20240501 19:56:54.333663 30698 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:54.333989 30698 tablet.cc:1620] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:54.334909 24450 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet e63440e636484ab481007ef81ceceaa3 (table rebalancer_dynamic_table [id=d2224d9fb54b44cebe21304c7b1df9f5]). Expected version 1 got 0
I20240501 19:56:54.337945 24451 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet e63440e636484ab481007ef81ceceaa3 (table rebalancer_dynamic_table [id=d2224d9fb54b44cebe21304c7b1df9f5]). Expected version 1 got 0
I20240501 19:56:54.337874 30881 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:54.339095 30881 tablet.cc:1620] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:56:54.340765 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=d2224d9fb54b44cebe21304c7b1df9f5] alter complete (version 1)
I20240501 19:56:54.341786 30696 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:54.342085 30696 tablet.cc:1620] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:54.365892 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:54.374146 27274 tablet_service.cc:1467] Processing CreateTablet for tablet ca1d950141e14ca68da6d8b3e8254552 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.374363 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 05cc3b1114a04788821eca2c6343f793 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.374471 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1d950141e14ca68da6d8b3e8254552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.374653 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cc3b1114a04788821eca2c6343f793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.375792 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 5515207fb0134f8da7f58f4cf730fae7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.375789 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 05cc3b1114a04788821eca2c6343f793 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.376026 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5515207fb0134f8da7f58f4cf730fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.376060 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cc3b1114a04788821eca2c6343f793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.377012 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 05cc3b1114a04788821eca2c6343f793 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.377274 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cc3b1114a04788821eca2c6343f793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.378334 30815 tablet_bootstrap.cc:492] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:54.378378 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 5515207fb0134f8da7f58f4cf730fae7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.378608 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5515207fb0134f8da7f58f4cf730fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.379284 30815 tablet_bootstrap.cc:654] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.379509 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 5515207fb0134f8da7f58f4cf730fae7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.379786 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5515207fb0134f8da7f58f4cf730fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.380707 27404 tablet_service.cc:1467] Processing CreateTablet for tablet ca1d950141e14ca68da6d8b3e8254552 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.380929 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1d950141e14ca68da6d8b3e8254552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.382532 27010 tablet_service.cc:1467] Processing CreateTablet for tablet ca1d950141e14ca68da6d8b3e8254552 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:54.382772 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1d950141e14ca68da6d8b3e8254552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.388818 30818 tablet_bootstrap.cc:492] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:54.389791 30818 tablet_bootstrap.cc:654] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.391731 30818 tablet_bootstrap.cc:492] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:54.391815 30818 ts_tablet_manager.cc:1367] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:54.392395 30818 raft_consensus.cc:348] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.392503 30818 raft_consensus.cc:374] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.392534 30818 raft_consensus.cc:729] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.392648 30818 consensus_queue.cc:260] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.392935 30818 ts_tablet_manager.cc:1398] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:54.393735 27031 raft_consensus.cc:1203] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:54.407487 30912 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:54.407729 30815 tablet_bootstrap.cc:492] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:54.407833 30815 ts_tablet_manager.cc:1367] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240501 19:56:54.408375 30815 raft_consensus.cc:348] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.408517 30815 raft_consensus.cc:374] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.408551 30815 raft_consensus.cc:729] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.408691 30815 consensus_queue.cc:260] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.409042 30815 ts_tablet_manager.cc:1398] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:54.409211 30815 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:54.409935 30912 tablet_bootstrap.cc:654] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.410177 30815 tablet_bootstrap.cc:654] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.413403 30815 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:54.413501 30815 ts_tablet_manager.cc:1367] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:54.413923 30815 raft_consensus.cc:348] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.414040 30815 raft_consensus.cc:374] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.414079 30815 raft_consensus.cc:729] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.414214 30815 consensus_queue.cc:260] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.414309 30913 tablet_bootstrap.cc:492] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:54.414513 30815 ts_tablet_manager.cc:1398] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:54.414661 30815 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:54.396034 30915 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:54.417528 30815 tablet_bootstrap.cc:654] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.418627 30913 tablet_bootstrap.cc:654] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.418990 30915 tablet_bootstrap.cc:654] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.419384 30912 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:54.419562 30912 ts_tablet_manager.cc:1367] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240501 19:56:54.420085 30912 raft_consensus.cc:348] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.420271 30912 raft_consensus.cc:374] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.420344 30912 raft_consensus.cc:729] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.420543 30912 consensus_queue.cc:260] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.421751 30815 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:54.421877 30815 ts_tablet_manager.cc:1367] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:54.422562 30815 raft_consensus.cc:348] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.423079 30815 raft_consensus.cc:374] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.423312 30815 raft_consensus.cc:729] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.423529 30815 consensus_queue.cc:260] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.425887 30815 ts_tablet_manager.cc:1398] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:54.426270 30913 tablet_bootstrap.cc:492] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:54.426411 30913 ts_tablet_manager.cc:1367] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:54.426909 30913 raft_consensus.cc:348] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.427117 30913 raft_consensus.cc:374] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.427217 30913 raft_consensus.cc:729] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.427414 30913 consensus_queue.cc:260] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.427958 30912 ts_tablet_manager.cc:1398] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240501 19:56:54.428597 30913 ts_tablet_manager.cc:1398] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:54.430138 30915 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:54.430294 30915 ts_tablet_manager.cc:1367] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240501 19:56:54.430843 30915 raft_consensus.cc:348] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.432058 30915 raft_consensus.cc:374] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.432197 30915 raft_consensus.cc:729] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.432628 27142 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.433830 30913 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:54.432968 30915 consensus_queue.cc:260] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.435308 30915 ts_tablet_manager.cc:1398] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:54.439389 27274 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.443006 27010 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:54.443372 27404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:54.445236 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:54.447486 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:54.457418 30698 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:54.457731 30698 tablet.cc:1620] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:54.458820 30913 tablet_bootstrap.cc:654] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.460462 30696 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:54.460762 30696 tablet.cc:1620] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:54.465636 30881 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:54.465917 30881 tablet.cc:1620] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:56:54.469069 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=d2224d9fb54b44cebe21304c7b1df9f5] alter complete (version 2)
I20240501 19:56:54.470975 30913 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:54.471139 30913 ts_tablet_manager.cc:1367] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240501 19:56:54.471132 30587 raft_consensus.cc:482] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:54.471220 30587 raft_consensus.cc:504] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.471562 30587 leader_election.cc:288] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:54.471604 30913 raft_consensus.cc:348] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.471717 30913 raft_consensus.cc:374] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.471752 30913 raft_consensus.cc:729] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.471894 30913 consensus_queue.cc:260] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.471946 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:54.472190 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:56:54.472420 27359 leader_election.cc:341] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:54.472887 30913 ts_tablet_manager.cc:1398] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:54.472875 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:54.472996 27276 raft_consensus.cc:2454] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:54.473042 30913 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:54.473474 27361 leader_election.cc:302] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:54.473686 30587 raft_consensus.cc:2781] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:54.473728 30587 raft_consensus.cc:482] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:54.473762 30587 raft_consensus.cc:3037] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.473878 30913 tablet_bootstrap.cc:654] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.474658 30587 raft_consensus.cc:504] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.474973 30587 leader_election.cc:288] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:54.475418 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
W20240501 19:56:54.475688 27359 leader_election.cc:341] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:54.476039 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:54.476132 27276 raft_consensus.cc:3037] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.477008 27276 raft_consensus.cc:2454] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:54.477547 27361 leader_election.cc:302] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:54.477999 30587 raft_consensus.cc:2781] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:54.478157 30587 raft_consensus.cc:686] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:54.478274 30587 consensus_queue.cc:237] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.480026 24453 catalog_manager.cc:5484] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:54.482517 30913 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:54.482605 30913 ts_tablet_manager.cc:1367] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:56:54.483412 30913 raft_consensus.cc:348] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.483523 30913 raft_consensus.cc:374] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.483556 30913 raft_consensus.cc:729] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.483649 30913 consensus_queue.cc:260] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.492501 30913 ts_tablet_manager.cc:1398] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:56:54.495416 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:54.495904 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:54.541009 24725 consensus_queue.cc:237] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:54.542217 27276 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.542317 27163 raft_consensus.cc:1261] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.542757 28709 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.542867 30378 consensus_queue.cc:1035] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.544502 30357 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:54.544880 27163 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:54.545070 27276 raft_consensus.cc:2932] T a4b240b461b548a087fcabdeb938df1e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:54.547678 24447 catalog_manager.cc:5484] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 6 to 7, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:54.550189 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b240b461b548a087fcabdeb938df1e with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:54.550568 30955 tablet_replica.cc:330] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:54.550908 30955 raft_consensus.cc:2229] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.551163 30955 raft_consensus.cc:2258] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.551687 30955 ts_tablet_manager.cc:1875] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.553573 30955 ts_tablet_manager.cc:1888] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:54.553661 30955 log.cc:1196] T a4b240b461b548a087fcabdeb938df1e P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/a4b240b461b548a087fcabdeb938df1e
I20240501 19:56:54.554221 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet a4b240b461b548a087fcabdeb938df1e (table rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]) successfully deleted
I20240501 19:56:54.554841 27425 raft_consensus.cc:1203] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:54.576774 29384 raft_consensus.cc:482] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:54.576875 29384 raft_consensus.cc:504] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.577214 29384 leader_election.cc:288] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:54.577972 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5515207fb0134f8da7f58f4cf730fae7" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:54.578099 24594 raft_consensus.cc:2454] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:54.578445 26965 leader_election.cc:302] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:54.579627 29384 raft_consensus.cc:2781] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:54.579685 29384 raft_consensus.cc:482] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:54.579718 29384 raft_consensus.cc:3037] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.580839 29384 raft_consensus.cc:504] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.581128 29384 leader_election.cc:288] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:54.581561 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5515207fb0134f8da7f58f4cf730fae7" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:54.581662 24594 raft_consensus.cc:3037] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.581609 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5515207fb0134f8da7f58f4cf730fae7" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:54.581741 27424 raft_consensus.cc:3037] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.582391 24594 raft_consensus.cc:2454] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:54.582679 27424 raft_consensus.cc:2454] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:54.582696 26965 leader_election.cc:302] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:54.582842 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5515207fb0134f8da7f58f4cf730fae7" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:54.582955 27425 raft_consensus.cc:2362] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1. Re-sending same reply.
I20240501 19:56:54.583601 29384 raft_consensus.cc:2781] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:54.583774 29384 raft_consensus.cc:686] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:54.583920 29384 consensus_queue.cc:237] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:54.586506 24453 catalog_manager.cc:5484] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:54.589972 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:54.597134 30698 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:54.597945 30696 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:54.598177 30881 tablet.cc:1722] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:54.602860 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=d2224d9fb54b44cebe21304c7b1df9f5] alter complete (version 3)
I20240501 19:56:54.651322 30587 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:54.651460 30587 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.651818 30587 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:54.652148 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cc3b1114a04788821eca2c6343f793" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:54.652148 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cc3b1114a04788821eca2c6343f793" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:54.652281 27031 raft_consensus.cc:2454] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:54.652280 27163 raft_consensus.cc:2454] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:54.652530 27359 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:54.652762 30587 raft_consensus.cc:2781] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:54.652828 30587 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:54.652866 30587 raft_consensus.cc:3037] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.653805 30587 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.654106 30587 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:54.654426 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cc3b1114a04788821eca2c6343f793" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:54.654418 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cc3b1114a04788821eca2c6343f793" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:54.654533 27031 raft_consensus.cc:3037] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.654533 27163 raft_consensus.cc:3037] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.655530 27163 raft_consensus.cc:2454] T 05cc3b1114a04788821eca2c6343f793 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:54.655530 27031 raft_consensus.cc:2454] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:54.655797 27361 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:54.656005 30587 raft_consensus.cc:2781] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:54.656121 30587 raft_consensus.cc:686] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:54.656255 30587 consensus_queue.cc:237] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.658241 24449 catalog_manager.cc:5484] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:54.679831 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.680097 27010 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:54.680513 27142 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.681447 27404 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:54.686322 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:54.689894 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:54.704035 24449 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:54.704195 24449 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:54.704903 27274 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.706344 24449 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet e63440e636484ab481007ef81ceceaa3
I20240501 19:56:54.706946 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:54.707665 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet e63440e636484ab481007ef81ceceaa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:54 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:54.707677 27404 tablet_service.cc:1514] Processing DeleteTablet for tablet e63440e636484ab481007ef81ceceaa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:54 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:54.707768 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet e63440e636484ab481007ef81ceceaa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:54 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:54.723095 31012 tablet_replica.cc:330] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:54.725668 31012 raft_consensus.cc:2229] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:54.725950 31012 raft_consensus.cc:2258] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.729120 31012 ts_tablet_manager.cc:1875] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:54.730255 31013 tablet_replica.cc:330] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:54.730463 31014 tablet_replica.cc:330] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:54.730726 31013 raft_consensus.cc:2229] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.730909 31014 raft_consensus.cc:2229] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.731119 31013 raft_consensus.cc:2258] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.731273 31014 raft_consensus.cc:2258] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.732266 31014 ts_tablet_manager.cc:1875] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:54.734359 31014 ts_tablet_manager.cc:1888] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:54.734779 31013 ts_tablet_manager.cc:1875] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:54.736845 31013 ts_tablet_manager.cc:1888] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:54.736918 31013 log.cc:1196] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/e63440e636484ab481007ef81ceceaa3
I20240501 19:56:54.737223 31013 ts_tablet_manager.cc:1909] T e63440e636484ab481007ef81ceceaa3 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:54.737608 31012 ts_tablet_manager.cc:1888] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:54.737686 31012 log.cc:1196] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/e63440e636484ab481007ef81ceceaa3
I20240501 19:56:54.737857 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet e63440e636484ab481007ef81ceceaa3 (table rebalancer_dynamic_table_new_name [id=d2224d9fb54b44cebe21304c7b1df9f5]) successfully deleted
I20240501 19:56:54.737922 31012 ts_tablet_manager.cc:1909] T e63440e636484ab481007ef81ceceaa3 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:54.738615 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet e63440e636484ab481007ef81ceceaa3 (table rebalancer_dynamic_table_new_name [id=d2224d9fb54b44cebe21304c7b1df9f5]) successfully deleted
I20240501 19:56:54.739666 31014 log.cc:1196] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/e63440e636484ab481007ef81ceceaa3
I20240501 19:56:54.739996 31014 ts_tablet_manager.cc:1909] T e63440e636484ab481007ef81ceceaa3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:54.740599 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet e63440e636484ab481007ef81ceceaa3 (table rebalancer_dynamic_table_new_name [id=d2224d9fb54b44cebe21304c7b1df9f5]) successfully deleted
I20240501 19:56:54.765352 29384 raft_consensus.cc:482] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:54.765457 29384 raft_consensus.cc:504] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:54.765799 29384 leader_election.cc:288] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:54.766112 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:54.766122 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:54.766287 27278 raft_consensus.cc:2379] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in current term 1: Already voted for candidate 4a6e54e86a9949e5970b42e8e32e7561 in this term.
I20240501 19:56:54.766651 26967 leader_election.cc:302] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610
I20240501 19:56:54.766937 29384 raft_consensus.cc:3037] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.768011 29384 raft_consensus.cc:2726] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:54.768030 24986 consensus_queue.cc:237] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:54.769173 27278 raft_consensus.cc:1261] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:54.769214 27163 raft_consensus.cc:1261] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:54.769574 30710 consensus_queue.cc:1035] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:54.769680 30481 consensus_queue.cc:1035] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:54.771323 30481 raft_consensus.cc:2932] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:54.773046 27278 raft_consensus.cc:2932] T 166bb900935040bd9f7723e822780e6c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:54.775593 24449 catalog_manager.cc:5484] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:54.778460 27404 tablet_service.cc:1514] Processing DeleteTablet for tablet 166bb900935040bd9f7723e822780e6c with delete_type TABLET_DATA_TOMBSTONED (TS 4a6e54e86a9949e5970b42e8e32e7561 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:54.779341 31013 tablet_replica.cc:330] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:54.779489 31013 raft_consensus.cc:2229] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.779614 31013 raft_consensus.cc:2258] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.780046 31013 ts_tablet_manager.cc:1875] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.781934 31013 ts_tablet_manager.cc:1888] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:56:54.782002 31013 log.cc:1196] T 166bb900935040bd9f7723e822780e6c P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/166bb900935040bd9f7723e822780e6c
I20240501 19:56:54.782502 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 166bb900935040bd9f7723e822780e6c (table rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]) successfully deleted
I20240501 19:56:54.785763 27163 raft_consensus.cc:2932] T 166bb900935040bd9f7723e822780e6c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:54.808746 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:54.813958 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 8a37bd433a3f427d9dae87770e6a24cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=09d665b487f447038ac05313a227521a]), partition=
I20240501 19:56:54.813979 27010 tablet_service.cc:1467] Processing CreateTablet for tablet 8a37bd433a3f427d9dae87770e6a24cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=09d665b487f447038ac05313a227521a]), partition=
I20240501 19:56:54.814008 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 8a37bd433a3f427d9dae87770e6a24cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=09d665b487f447038ac05313a227521a]), partition=
I20240501 19:56:54.814231 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a37bd433a3f427d9dae87770e6a24cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.814230 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a37bd433a3f427d9dae87770e6a24cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.814229 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a37bd433a3f427d9dae87770e6a24cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.815984 30818 tablet_bootstrap.cc:492] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:54.816471 30913 tablet_bootstrap.cc:492] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:54.816649 31049 tablet_bootstrap.cc:492] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:54.816999 30818 tablet_bootstrap.cc:654] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.817368 30913 tablet_bootstrap.cc:654] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.817628 31049 tablet_bootstrap.cc:654] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.818439 30818 tablet_bootstrap.cc:492] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:54.818529 30818 ts_tablet_manager.cc:1367] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:54.819005 30818 raft_consensus.cc:348] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:54.819154 30818 raft_consensus.cc:374] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.819192 30818 raft_consensus.cc:729] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.819325 30818 consensus_queue.cc:260] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:54.819650 30818 ts_tablet_manager.cc:1398] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:54.820385 30913 tablet_bootstrap.cc:492] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:54.820447 30913 ts_tablet_manager.cc:1367] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:54.820777 30913 raft_consensus.cc:348] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:54.820850 30913 raft_consensus.cc:374] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.820870 30913 raft_consensus.cc:729] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.820971 30913 consensus_queue.cc:260] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:54.821275 30913 ts_tablet_manager.cc:1398] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:54.822162 31049 tablet_bootstrap.cc:492] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:54.822252 31049 ts_tablet_manager.cc:1367] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:54.822754 31049 raft_consensus.cc:348] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:54.822851 31049 raft_consensus.cc:374] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.822885 31049 raft_consensus.cc:729] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.823055 31049 consensus_queue.cc:260] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:54.823467 31049 ts_tablet_manager.cc:1398] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.019335 30967 consensus_queue.cc:1035] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.022310 30967 consensus_queue.cc:1035] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.050563 30935 consensus_queue.cc:1035] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.054230 30935 consensus_queue.cc:1035] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.177626 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:55.184259 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 45cabeeee9ad4ec6830a212e93fd2948 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.184592 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cabeeee9ad4ec6830a212e93fd2948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.184948 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 99737b8809b345ceb15ca74fc313bf4d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.185099 27404 tablet_service.cc:1467] Processing CreateTablet for tablet 45cabeeee9ad4ec6830a212e93fd2948 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.185575 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 99737b8809b345ceb15ca74fc313bf4d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.185573 27405 tablet_service.cc:1467] Processing CreateTablet for tablet fe1f00507b9f48d786a378a15baa4d5b (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.185705 27273 tablet_service.cc:1467] Processing CreateTablet for tablet fe1f00507b9f48d786a378a15baa4d5b (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.185819 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99737b8809b345ceb15ca74fc313bf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.185942 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1f00507b9f48d786a378a15baa4d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.187075 31086 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:55.185181 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99737b8809b345ceb15ca74fc313bf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.187538 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 99737b8809b345ceb15ca74fc313bf4d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.187692 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1f00507b9f48d786a378a15baa4d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.187793 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99737b8809b345ceb15ca74fc313bf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.187876 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cabeeee9ad4ec6830a212e93fd2948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.189766 30818 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:55.190544 30818 tablet_bootstrap.cc:654] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.191200 31086 tablet_bootstrap.cc:654] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.192291 31087 tablet_bootstrap.cc:492] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:55.192889 27012 tablet_service.cc:1467] Processing CreateTablet for tablet fe1f00507b9f48d786a378a15baa4d5b (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.193177 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1f00507b9f48d786a378a15baa4d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.193208 31087 tablet_bootstrap.cc:654] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.193192 27010 tablet_service.cc:1467] Processing CreateTablet for tablet 45cabeeee9ad4ec6830a212e93fd2948 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.193845 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cabeeee9ad4ec6830a212e93fd2948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.195441 30913 tablet_bootstrap.cc:492] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:55.196949 31086 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:55.197037 31086 ts_tablet_manager.cc:1367] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 19:56:55.197064 31087 tablet_bootstrap.cc:492] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:55.197122 30913 tablet_bootstrap.cc:654] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.197160 31087 ts_tablet_manager.cc:1367] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:55.197448 31086 raft_consensus.cc:348] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.197544 31086 raft_consensus.cc:374] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.197577 31086 raft_consensus.cc:729] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.197712 31086 consensus_queue.cc:260] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.197896 31087 raft_consensus.cc:348] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.198002 31087 raft_consensus.cc:374] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.198026 31086 ts_tablet_manager.cc:1398] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:55.198036 31087 raft_consensus.cc:729] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.198184 31087 consensus_queue.cc:260] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.198544 31087 ts_tablet_manager.cc:1398] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.199672 30913 tablet_bootstrap.cc:492] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:55.199733 31087 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:55.199777 30913 ts_tablet_manager.cc:1367] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:55.200309 30913 raft_consensus.cc:348] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.200449 30913 raft_consensus.cc:374] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.200486 30913 raft_consensus.cc:729] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.200619 30913 consensus_queue.cc:260] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.200910 30913 ts_tablet_manager.cc:1398] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.200990 31086 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:55.201246 31087 tablet_bootstrap.cc:654] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.201922 30913 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:55.201946 31086 tablet_bootstrap.cc:654] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.203652 31087 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:55.203740 31087 ts_tablet_manager.cc:1367] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:55.204242 31087 raft_consensus.cc:348] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.204380 31087 raft_consensus.cc:374] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.204443 31087 raft_consensus.cc:729] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.204576 31087 consensus_queue.cc:260] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.204694 31086 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:55.204699 30913 tablet_bootstrap.cc:654] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.204764 31086 ts_tablet_manager.cc:1367] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:56:55.204928 31087 ts_tablet_manager.cc:1398] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:55.205096 31087 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:55.205121 31086 raft_consensus.cc:348] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.205194 31086 raft_consensus.cc:374] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.205219 31086 raft_consensus.cc:729] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.205319 31086 consensus_queue.cc:260] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.205572 31086 ts_tablet_manager.cc:1398] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:55.205700 31086 tablet_bootstrap.cc:492] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:55.205849 30913 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:55.205927 30913 ts_tablet_manager.cc:1367] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:55.206092 31087 tablet_bootstrap.cc:654] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.206311 31086 tablet_bootstrap.cc:654] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.206450 30913 raft_consensus.cc:348] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.206580 30913 raft_consensus.cc:374] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.206629 30913 raft_consensus.cc:729] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.206784 30913 consensus_queue.cc:260] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.207300 30913 ts_tablet_manager.cc:1398] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.208429 30818 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:55.208554 30818 ts_tablet_manager.cc:1367] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 19:56:55.208978 30818 raft_consensus.cc:348] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.209079 30818 raft_consensus.cc:374] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.209110 30818 raft_consensus.cc:729] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.209239 30818 consensus_queue.cc:260] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.209512 30818 ts_tablet_manager.cc:1398] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.210529 31087 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:55.210624 31087 ts_tablet_manager.cc:1367] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 19:56:55.210819 31086 tablet_bootstrap.cc:492] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:55.210917 31086 ts_tablet_manager.cc:1367] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 19:56:55.211102 31087 raft_consensus.cc:348] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.211323 31087 raft_consensus.cc:374] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.211421 31087 raft_consensus.cc:729] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.211452 31086 raft_consensus.cc:348] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.211565 31086 raft_consensus.cc:374] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.211602 31086 raft_consensus.cc:729] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.211728 31087 consensus_queue.cc:260] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.211777 31086 consensus_queue.cc:260] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.212059 31086 ts_tablet_manager.cc:1398] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.212621 29464 raft_consensus.cc:482] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:55.212709 29464 raft_consensus.cc:504] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.213075 29464 leader_election.cc:288] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:55.214998 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:55.215019 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:55.215137 27163 raft_consensus.cc:2454] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:55.215154 27425 raft_consensus.cc:2454] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:55.215288 31087 ts_tablet_manager.cc:1398] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:56:55.215467 27229 leader_election.cc:302] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:55.215795 29464 raft_consensus.cc:2781] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:55.215847 29464 raft_consensus.cc:482] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:55.215878 29464 raft_consensus.cc:3037] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.216745 29464 raft_consensus.cc:504] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.217111 29464 leader_election.cc:288] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:55.217487 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:55.217497 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:55.217607 27425 raft_consensus.cc:3037] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.217614 27163 raft_consensus.cc:3037] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.218531 27425 raft_consensus.cc:2454] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:55.218654 27163 raft_consensus.cc:2454] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:55.218910 27229 leader_election.cc:302] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:55.219106 29464 raft_consensus.cc:2781] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:55.219223 29464 raft_consensus.cc:686] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:55.219465 29464 consensus_queue.cc:237] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.221963 24453 catalog_manager.cc:5484] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:55.222965 30378 raft_consensus.cc:482] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:55.223101 30378 raft_consensus.cc:504] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:55.223490 30378 leader_election.cc:288] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:55.223794 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a37bd433a3f427d9dae87770e6a24cc" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:55.223889 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a37bd433a3f427d9dae87770e6a24cc" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:55.223971 27032 raft_consensus.cc:2454] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:55.224018 27163 raft_consensus.cc:2454] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:56:55.224334 24660 leader_election.cc:302] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:55.224489 30378 raft_consensus.cc:2781] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:55.224541 30378 raft_consensus.cc:482] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:55.224577 30378 raft_consensus.cc:3037] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.225566 30378 raft_consensus.cc:504] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:55.225890 30378 leader_election.cc:288] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:55.226204 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a37bd433a3f427d9dae87770e6a24cc" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:55.226233 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a37bd433a3f427d9dae87770e6a24cc" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:55.226322 27163 raft_consensus.cc:3037] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.226315 27032 raft_consensus.cc:3037] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.227164 27032 raft_consensus.cc:2454] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:55.227325 27163 raft_consensus.cc:2454] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:56:55.227416 24660 leader_election.cc:302] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:55.227622 30378 raft_consensus.cc:2781] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:55.227833 30378 raft_consensus.cc:686] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:55.227978 30378 consensus_queue.cc:237] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:55.229991 24453 catalog_manager.cc:5484] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:55.247121 27142 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.250633 27010 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:55.254523 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:55.256330 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:56:55.264639 27273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:55.268527 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:55.268963 30587 consensus_queue.cc:1035] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.270004 30935 consensus_queue.cc:1035] T 05cc3b1114a04788821eca2c6343f793 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.280409 29464 raft_consensus.cc:482] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:55.280519 29464 raft_consensus.cc:504] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.280848 29464 leader_election.cc:288] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:55.281666 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cabeeee9ad4ec6830a212e93fd2948" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:55.281795 27031 raft_consensus.cc:2454] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:55.282223 27227 leader_election.cc:302] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:55.282389 29464 raft_consensus.cc:2781] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:55.282445 29464 raft_consensus.cc:482] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:55.282485 29464 raft_consensus.cc:3037] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.282464 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cabeeee9ad4ec6830a212e93fd2948" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:55.282578 27425 raft_consensus.cc:2454] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:55.283641 29464 raft_consensus.cc:504] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.283973 29464 leader_election.cc:288] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:55.284571 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cabeeee9ad4ec6830a212e93fd2948" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:55.284674 27031 raft_consensus.cc:3037] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.285636 27031 raft_consensus.cc:2454] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:55.285909 27227 leader_election.cc:302] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:55.286067 29464 raft_consensus.cc:2781] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:55.286126 29464 raft_consensus.cc:686] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:55.286238 29464 consensus_queue.cc:237] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.286897 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cabeeee9ad4ec6830a212e93fd2948" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:55.287003 27425 raft_consensus.cc:3037] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.287999 27425 raft_consensus.cc:2454] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:55.288385 24453 catalog_manager.cc:5484] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:55.288525 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:55.308156 27404 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.329375 31085 raft_consensus.cc:482] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:55.329524 31085 raft_consensus.cc:504] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.329998 31085 leader_election.cc:288] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:55.330487 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1f00507b9f48d786a378a15baa4d5b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:55.330638 27032 raft_consensus.cc:2454] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:55.331367 27227 leader_election.cc:302] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:55.332252 31085 raft_consensus.cc:2781] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:55.332331 31085 raft_consensus.cc:482] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:55.332372 31085 raft_consensus.cc:3037] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.333372 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1f00507b9f48d786a378a15baa4d5b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:55.333500 27425 raft_consensus.cc:2454] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:55.333472 31085 raft_consensus.cc:504] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.333779 31085 leader_election.cc:288] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:55.334136 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1f00507b9f48d786a378a15baa4d5b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:55.334275 27032 raft_consensus.cc:3037] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.334286 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1f00507b9f48d786a378a15baa4d5b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:55.334380 27425 raft_consensus.cc:3037] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.335318 27032 raft_consensus.cc:2454] T fe1f00507b9f48d786a378a15baa4d5b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:55.335318 27425 raft_consensus.cc:2454] T fe1f00507b9f48d786a378a15baa4d5b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:55.335702 27229 leader_election.cc:302] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:55.335983 31085 raft_consensus.cc:2781] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:55.336103 31085 raft_consensus.cc:686] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:55.336246 31085 consensus_queue.cc:237] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:55.338788 24453 catalog_manager.cc:5484] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:55.354207 27278 consensus_queue.cc:237] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:56:55.355713 24725 raft_consensus.cc:1261] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:55.355917 27425 raft_consensus.cc:1261] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:55.356104 31085 consensus_queue.cc:1035] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:55.356292 29464 consensus_queue.cc:1035] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:55.358160 31142 raft_consensus.cc:2932] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:55.358474 27425 raft_consensus.cc:2932] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:55.358501 24725 raft_consensus.cc:2932] T 536a3171eab9403eab069fa86c70a8b5 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:55.361227 24453 catalog_manager.cc:5484] T 536a3171eab9403eab069fa86c70a8b5 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:56:55.364733 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 536a3171eab9403eab069fa86c70a8b5 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:55.365099 31147 tablet_replica.cc:330] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:55.365273 31147 raft_consensus.cc:2229] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.365430 31147 raft_consensus.cc:2258] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.366037 31147 ts_tablet_manager.cc:1875] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:55.368535 31147 ts_tablet_manager.cc:1888] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:56:55.368506 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:55.368705 31147 log.cc:1196] T 536a3171eab9403eab069fa86c70a8b5 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/536a3171eab9403eab069fa86c70a8b5
I20240501 19:56:55.369783 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 536a3171eab9403eab069fa86c70a8b5 (table rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]) successfully deleted
I20240501 19:56:55.373206 27163 raft_consensus.cc:1261] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:55.373314 27032 raft_consensus.cc:1261] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:55.373792 30378 consensus_queue.cc:1035] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.374258 31099 consensus_queue.cc:1035] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.376278 24445 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 8a37bd433a3f427d9dae87770e6a24cc (table rebalancer_dynamic_table [id=09d665b487f447038ac05313a227521a]). Expected version 1 got 0
I20240501 19:56:55.376777 24452 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 8a37bd433a3f427d9dae87770e6a24cc (table rebalancer_dynamic_table [id=09d665b487f447038ac05313a227521a]). Expected version 1 got 0
I20240501 19:56:55.378579 31158 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:55.378700 24445 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 8a37bd433a3f427d9dae87770e6a24cc (table rebalancer_dynamic_table [id=09d665b487f447038ac05313a227521a]). Expected version 1 got 0
I20240501 19:56:55.379299 31159 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:55.379621 31158 tablet.cc:1620] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:55.379632 31159 tablet.cc:1620] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:55.383329 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=09d665b487f447038ac05313a227521a] alter complete (version 1)
I20240501 19:56:55.387578 31157 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:55.388141 31157 tablet.cc:1620] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:55.478415 27010 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:55.478977 27142 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.485738 27273 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:55.487262 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:55.489897 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:55.491827 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:56:55.496048 31157 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:55.496356 31157 tablet.cc:1620] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:56:55.497736 31158 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:55.498008 31158 tablet.cc:1620] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:55.498418 31159 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:55.498679 31159 tablet.cc:1620] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:55.507856 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=09d665b487f447038ac05313a227521a] alter complete (version 2)
I20240501 19:56:55.509059 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:55.527995 27404 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.533202 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:55.578186 24725 consensus_queue.cc:237] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.579437 27163 raft_consensus.cc:1261] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:55.579456 24595 raft_consensus.cc:1261] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:55.579838 30378 consensus_queue.cc:1035] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:55.580152 30378 consensus_queue.cc:1035] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:55.583740 31099 raft_consensus.cc:2932] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:55.585311 27163 raft_consensus.cc:2932] T e75026e7e70548e190f81bf79fb30926 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:55.587675 24450 catalog_manager.cc:5484] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:55.589414 24595 raft_consensus.cc:2932] T e75026e7e70548e190f81bf79fb30926 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:55.590210 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet e75026e7e70548e190f81bf79fb30926 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:55.590981 31147 tablet_replica.cc:330] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:55.591153 31147 raft_consensus.cc:2229] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.591312 31147 raft_consensus.cc:2258] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.591826 31147 ts_tablet_manager.cc:1875] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:55.593408 31147 ts_tablet_manager.cc:1888] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:55.593461 31147 log.cc:1196] T e75026e7e70548e190f81bf79fb30926 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/e75026e7e70548e190f81bf79fb30926
I20240501 19:56:55.593906 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet e75026e7e70548e190f81bf79fb30926 (table rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]) successfully deleted
I20240501 19:56:55.612696 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:55.618052 31157 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:55.618538 31159 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:55.623598 31142 consensus_queue.cc:1035] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.624231 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=09d665b487f447038ac05313a227521a] alter complete (version 3)
I20240501 19:56:55.624992 31158 tablet.cc:1722] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:55.628965 31142 consensus_queue.cc:1035] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.688725 27142 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.696835 27010 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:55.697819 27273 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:55.710103 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:56:55.710476 27404 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.713901 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:55.715620 24837 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:55.716226 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:55.728379 29464 consensus_queue.cc:1035] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.728657 24449 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:55.728790 24449 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:55.731312 24449 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 8a37bd433a3f427d9dae87770e6a24cc
I20240501 19:56:55.731683 29464 consensus_queue.cc:1035] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.731851 27010 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a37bd433a3f427d9dae87770e6a24cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:55 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:55.732214 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a37bd433a3f427d9dae87770e6a24cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:55.733635 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a37bd433a3f427d9dae87770e6a24cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:55 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:56:55.734879 31279 tablet_replica.cc:330] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:56:55.735118 31279 raft_consensus.cc:2229] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:55.735383 31279 raft_consensus.cc:2258] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.738636 31276 tablet_replica.cc:330] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:55.740000 31276 raft_consensus.cc:2229] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.740165 31276 raft_consensus.cc:2258] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.741094 31276 ts_tablet_manager.cc:1875] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:55.743434 31276 ts_tablet_manager.cc:1888] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:55.743530 31276 log.cc:1196] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/8a37bd433a3f427d9dae87770e6a24cc
I20240501 19:56:55.743813 31276 ts_tablet_manager.cc:1909] T 8a37bd433a3f427d9dae87770e6a24cc P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:55.744591 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 8a37bd433a3f427d9dae87770e6a24cc (table rebalancer_dynamic_table_new_name [id=09d665b487f447038ac05313a227521a]) successfully deleted
I20240501 19:56:55.747449 31279 ts_tablet_manager.cc:1875] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:55.749578 31279 ts_tablet_manager.cc:1888] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:55.749706 31279 log.cc:1196] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a37bd433a3f427d9dae87770e6a24cc
I20240501 19:56:55.749998 31279 ts_tablet_manager.cc:1909] T 8a37bd433a3f427d9dae87770e6a24cc P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:56:55.752501 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 8a37bd433a3f427d9dae87770e6a24cc (table rebalancer_dynamic_table_new_name [id=09d665b487f447038ac05313a227521a]) successfully deleted
I20240501 19:56:55.759601 31277 tablet_replica.cc:330] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:55.759764 31277 raft_consensus.cc:2229] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.759884 31277 raft_consensus.cc:2258] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.760421 31277 ts_tablet_manager.cc:1875] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:55.762550 31277 ts_tablet_manager.cc:1888] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:55.762640 31277 log.cc:1196] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/8a37bd433a3f427d9dae87770e6a24cc
I20240501 19:56:55.762919 31277 ts_tablet_manager.cc:1909] T 8a37bd433a3f427d9dae87770e6a24cc P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:55.763598 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 8a37bd433a3f427d9dae87770e6a24cc (table rebalancer_dynamic_table_new_name [id=09d665b487f447038ac05313a227521a]) successfully deleted
I20240501 19:56:55.802001 24857 consensus_queue.cc:237] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:55.803309 27278 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:55.803346 24725 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:55.803727 31297 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:55.803889 30151 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:55.805493 30151 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:55.806047 27278 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:55.808279 24725 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:55.808760 24453 catalog_manager.cc:5484] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:55.810768 31142 consensus_queue.cc:1035] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.811415 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 9acd3c8577694e799d3f67a0f2f6adcc with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:55.811731 31304 tablet_replica.cc:330] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:55.811913 31304 raft_consensus.cc:2229] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.812112 31304 raft_consensus.cc:2258] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.812625 31304 ts_tablet_manager.cc:1875] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:55.814816 31304 ts_tablet_manager.cc:1888] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:55.814951 31304 log.cc:1196] T 9acd3c8577694e799d3f67a0f2f6adcc P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/9acd3c8577694e799d3f67a0f2f6adcc
I20240501 19:56:55.815646 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 9acd3c8577694e799d3f67a0f2f6adcc (table rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]) successfully deleted
I20240501 19:56:55.815835 29464 consensus_queue.cc:1035] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:55.832901 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:55.838150 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 7b41045eecf144bc9acdd963622c4c30 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70b90d9d6fc34e6fa16ca554a9cb411a]), partition=
I20240501 19:56:55.838246 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 7b41045eecf144bc9acdd963622c4c30 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70b90d9d6fc34e6fa16ca554a9cb411a]), partition=
I20240501 19:56:55.838310 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 7b41045eecf144bc9acdd963622c4c30 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70b90d9d6fc34e6fa16ca554a9cb411a]), partition=
I20240501 19:56:55.838428 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b41045eecf144bc9acdd963622c4c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.838521 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b41045eecf144bc9acdd963622c4c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.838529 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b41045eecf144bc9acdd963622c4c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.840986 31314 tablet_bootstrap.cc:492] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:55.841017 31315 tablet_bootstrap.cc:492] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:55.841246 31316 tablet_bootstrap.cc:492] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:55.841892 31315 tablet_bootstrap.cc:654] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.842182 31314 tablet_bootstrap.cc:654] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.843667 31315 tablet_bootstrap.cc:492] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:55.843717 31316 tablet_bootstrap.cc:654] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.843765 31315 ts_tablet_manager.cc:1367] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:55.844305 31315 raft_consensus.cc:348] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.844508 31315 raft_consensus.cc:374] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.844565 31315 raft_consensus.cc:729] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.844717 31315 consensus_queue.cc:260] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.845032 31315 ts_tablet_manager.cc:1398] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.845398 31316 tablet_bootstrap.cc:492] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:55.845494 31316 ts_tablet_manager.cc:1367] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:55.845688 31314 tablet_bootstrap.cc:492] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:55.845911 31314 ts_tablet_manager.cc:1367] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:55.846544 31314 raft_consensus.cc:348] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.847235 31316 raft_consensus.cc:348] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.847400 31314 raft_consensus.cc:374] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.847488 31316 raft_consensus.cc:374] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.847565 31316 raft_consensus.cc:729] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.847642 31314 raft_consensus.cc:729] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.847885 31316 consensus_queue.cc:260] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.848064 31314 consensus_queue.cc:260] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.848333 31314 ts_tablet_manager.cc:1398] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:55.849304 31316 ts_tablet_manager.cc:1398] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:55.873095 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:55.881024 24837 tablet_service.cc:1467] Processing CreateTablet for tablet a8020390483c4fd98265cbe7d8012435 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.881348 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8020390483c4fd98265cbe7d8012435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.881294 27010 tablet_service.cc:1467] Processing CreateTablet for tablet ec85cb24f1c445d9a853cabd2fc41aa0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.881593 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec85cb24f1c445d9a853cabd2fc41aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.881616 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 97f518e4e32644dc8a4af48941f85f55 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.881831 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f518e4e32644dc8a4af48941f85f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.882316 27273 tablet_service.cc:1467] Processing CreateTablet for tablet ec85cb24f1c445d9a853cabd2fc41aa0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.882584 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec85cb24f1c445d9a853cabd2fc41aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.883653 31314 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:55.884608 31314 tablet_bootstrap.cc:654] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.884560 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 97f518e4e32644dc8a4af48941f85f55 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.884824 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f518e4e32644dc8a4af48941f85f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.885061 27404 tablet_service.cc:1467] Processing CreateTablet for tablet ec85cb24f1c445d9a853cabd2fc41aa0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.885313 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec85cb24f1c445d9a853cabd2fc41aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.887727 27142 tablet_service.cc:1467] Processing CreateTablet for tablet a8020390483c4fd98265cbe7d8012435 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.887993 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8020390483c4fd98265cbe7d8012435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.890239 31316 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:55.887730 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 97f518e4e32644dc8a4af48941f85f55 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.890617 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f518e4e32644dc8a4af48941f85f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.891240 31316 tablet_bootstrap.cc:654] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.893128 31316 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:55.893211 31316 ts_tablet_manager.cc:1367] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:55.893671 31316 raft_consensus.cc:348] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.893781 31316 raft_consensus.cc:374] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.893824 31316 raft_consensus.cc:729] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.893944 31316 consensus_queue.cc:260] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.894150 27273 tablet_service.cc:1467] Processing CreateTablet for tablet a8020390483c4fd98265cbe7d8012435 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a567d922d3d24bb3ad0a2a92c5c09b2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.894223 31316 ts_tablet_manager.cc:1398] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.894409 31316 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:55.894419 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8020390483c4fd98265cbe7d8012435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.898962 31351 tablet_bootstrap.cc:492] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:55.900244 31350 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:55.900626 31316 tablet_bootstrap.cc:654] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.902179 31351 tablet_bootstrap.cc:654] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.902438 31353 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:55.903427 31353 tablet_bootstrap.cc:654] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.904106 31350 tablet_bootstrap.cc:654] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.905836 31316 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:55.905985 31316 ts_tablet_manager.cc:1367] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:55.906265 31351 tablet_bootstrap.cc:492] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:55.906394 31351 ts_tablet_manager.cc:1367] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:55.906467 31316 raft_consensus.cc:348] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.906606 31316 raft_consensus.cc:374] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.906682 31316 raft_consensus.cc:729] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.906834 31316 consensus_queue.cc:260] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.907152 31316 ts_tablet_manager.cc:1398] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.908123 31314 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:55.908221 31314 ts_tablet_manager.cc:1367] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 19:56:55.908711 31314 raft_consensus.cc:348] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.908819 31314 raft_consensus.cc:374] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.908852 31314 raft_consensus.cc:729] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.908964 31314 consensus_queue.cc:260] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.909260 31314 ts_tablet_manager.cc:1398] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.911787 31353 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:55.911880 31353 ts_tablet_manager.cc:1367] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:55.912528 31353 raft_consensus.cc:348] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.912701 31353 raft_consensus.cc:374] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.912776 31353 raft_consensus.cc:729] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.912986 31353 consensus_queue.cc:260] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.913892 31353 ts_tablet_manager.cc:1398] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:55.914310 31353 tablet_bootstrap.cc:492] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:55.915885 31350 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:55.916028 31350 ts_tablet_manager.cc:1367] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:56:55.916538 31350 raft_consensus.cc:348] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.916723 31350 raft_consensus.cc:374] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.916806 31350 raft_consensus.cc:729] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.917022 31350 consensus_queue.cc:260] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.917378 31350 ts_tablet_manager.cc:1398] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.919018 31350 tablet_bootstrap.cc:492] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:55.919483 31351 raft_consensus.cc:348] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:55.919615 31351 raft_consensus.cc:374] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.920253 31351 raft_consensus.cc:729] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.920318 31350 tablet_bootstrap.cc:654] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.920608 31351 consensus_queue.cc:260] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:55.920934 31351 ts_tablet_manager.cc:1398] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240501 19:56:55.921080 31353 tablet_bootstrap.cc:654] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.921121 31351 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:55.922286 31351 tablet_bootstrap.cc:654] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.922297 31353 tablet_bootstrap.cc:492] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:55.922398 31353 ts_tablet_manager.cc:1367] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:55.922822 31350 tablet_bootstrap.cc:492] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:55.922900 31350 ts_tablet_manager.cc:1367] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:55.922859 31353 raft_consensus.cc:348] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:55.922992 31353 raft_consensus.cc:374] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.923074 31353 raft_consensus.cc:729] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.923192 31353 consensus_queue.cc:260] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:55.923378 31350 raft_consensus.cc:348] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:55.923476 31350 raft_consensus.cc:374] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.923491 31353 ts_tablet_manager.cc:1398] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:55.923508 31350 raft_consensus.cc:729] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.923611 31350 consensus_queue.cc:260] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:55.923878 31350 ts_tablet_manager.cc:1398] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:55.924484 31351 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:55.924559 31351 ts_tablet_manager.cc:1367] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:55.924996 31351 raft_consensus.cc:348] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.925146 31351 raft_consensus.cc:374] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.925220 31351 raft_consensus.cc:729] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.925390 31351 consensus_queue.cc:260] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.925761 31351 ts_tablet_manager.cc:1398] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.936365 31297 raft_consensus.cc:482] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:55.936488 31297 raft_consensus.cc:504] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.936890 31297 leader_election.cc:288] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:55.937192 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b41045eecf144bc9acdd963622c4c30" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:55.937340 24595 raft_consensus.cc:2454] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:55.937458 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b41045eecf144bc9acdd963622c4c30" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:55.937605 27162 raft_consensus.cc:2454] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:55.937847 24791 leader_election.cc:302] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:55.938210 31297 raft_consensus.cc:2781] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:55.938264 31297 raft_consensus.cc:482] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:55.938310 31297 raft_consensus.cc:3037] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.939527 31297 raft_consensus.cc:504] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.939882 31297 leader_election.cc:288] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:55.940388 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b41045eecf144bc9acdd963622c4c30" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:55.940515 24595 raft_consensus.cc:3037] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.941304 24595 raft_consensus.cc:2454] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:55.941529 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b41045eecf144bc9acdd963622c4c30" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:55.941582 24791 leader_election.cc:302] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:55.941630 27162 raft_consensus.cc:3037] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.941818 31297 raft_consensus.cc:2781] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:55.941895 31297 raft_consensus.cc:686] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:55.942005 31297 consensus_queue.cc:237] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:55.942616 27162 raft_consensus.cc:2454] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:55.944056 24445 catalog_manager.cc:5484] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:55.959163 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:55.960915 27143 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.967450 27012 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:55.968022 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:55.968757 27273 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:55.972545 27404 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.981254 30967 raft_consensus.cc:482] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:55.981360 30967 raft_consensus.cc:504] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.981740 30967 leader_election.cc:288] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:55.982398 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f518e4e32644dc8a4af48941f85f55" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:55.982529 27162 raft_consensus.cc:2454] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:55.982789 26967 leader_election.cc:302] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:56:55.982798 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f518e4e32644dc8a4af48941f85f55" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:55.982913 27425 raft_consensus.cc:2454] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:55.982944 30967 raft_consensus.cc:2781] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:55.982981 30967 raft_consensus.cc:482] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:55.983014 30967 raft_consensus.cc:3037] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.984174 30967 raft_consensus.cc:504] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.984512 30967 leader_election.cc:288] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:55.984738 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:55.985329 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f518e4e32644dc8a4af48941f85f55" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:55.985320 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f518e4e32644dc8a4af48941f85f55" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:55.985426 27162 raft_consensus.cc:3037] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.985436 27425 raft_consensus.cc:3037] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.986352 27425 raft_consensus.cc:2454] T 97f518e4e32644dc8a4af48941f85f55 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:55.986352 27162 raft_consensus.cc:2454] T 97f518e4e32644dc8a4af48941f85f55 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:55.987095 26967 leader_election.cc:302] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:55.987479 30967 raft_consensus.cc:2781] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:55.987694 30967 raft_consensus.cc:686] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:55.987821 30967 consensus_queue.cc:237] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:55.993265 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:55.996739 24445 catalog_manager.cc:5484] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:56.018092 29464 raft_consensus.cc:482] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:56.018218 29464 raft_consensus.cc:504] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.018625 29464 leader_election.cc:288] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:56.018915 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec85cb24f1c445d9a853cabd2fc41aa0" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:56.019066 27032 raft_consensus.cc:2454] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:56.019008 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec85cb24f1c445d9a853cabd2fc41aa0" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:56.019155 27425 raft_consensus.cc:2454] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:56.019412 27229 leader_election.cc:302] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:56.019562 29464 raft_consensus.cc:2781] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:56.019618 29464 raft_consensus.cc:482] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:56.019652 29464 raft_consensus.cc:3037] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.020695 29464 raft_consensus.cc:504] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.021030 29464 leader_election.cc:288] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:56.021281 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec85cb24f1c445d9a853cabd2fc41aa0" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:56.021399 27425 raft_consensus.cc:3037] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.021409 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec85cb24f1c445d9a853cabd2fc41aa0" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:56.021508 27032 raft_consensus.cc:3037] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.022419 27425 raft_consensus.cc:2454] T ec85cb24f1c445d9a853cabd2fc41aa0 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:56.022496 27032 raft_consensus.cc:2454] T ec85cb24f1c445d9a853cabd2fc41aa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:56.022874 27229 leader_election.cc:302] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:56.023236 29464 raft_consensus.cc:2781] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:56.023327 29464 raft_consensus.cc:686] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:56.023464 29464 consensus_queue.cc:237] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.025516 24445 catalog_manager.cc:5484] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:56.047713 24595 consensus_queue.cc:237] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.048949 27278 raft_consensus.cc:1261] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:56.049048 24725 raft_consensus.cc:1261] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:56.049403 30811 consensus_queue.cc:1035] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:56.049603 31091 consensus_queue.cc:1035] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:56.050889 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:56.051748 30811 raft_consensus.cc:2932] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.053681 27278 raft_consensus.cc:2932] T 0ed4b95e555a46c5976ca9e084737ea1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.055421 24725 raft_consensus.cc:2932] T 0ed4b95e555a46c5976ca9e084737ea1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.057148 24445 catalog_manager.cc:5484] T 0ed4b95e555a46c5976ca9e084737ea1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.060111 27162 raft_consensus.cc:1261] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:56.060556 30151 consensus_queue.cc:1035] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.060868 24595 raft_consensus.cc:1261] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:56.060575 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed4b95e555a46c5976ca9e084737ea1 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:56.061237 31147 tablet_replica.cc:330] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:56.061381 31147 raft_consensus.cc:2229] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.061524 31147 raft_consensus.cc:2258] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.061836 30151 consensus_queue.cc:1035] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.062045 31147 ts_tablet_manager.cc:1875] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:56.064114 31147 ts_tablet_manager.cc:1888] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:56.064177 31147 log.cc:1196] T 0ed4b95e555a46c5976ca9e084737ea1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/0ed4b95e555a46c5976ca9e084737ea1
I20240501 19:56:56.064335 24448 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 7b41045eecf144bc9acdd963622c4c30 (table rebalancer_dynamic_table [id=70b90d9d6fc34e6fa16ca554a9cb411a]). Expected version 1 got 0
I20240501 19:56:56.064539 31403 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:56.065254 31403 tablet.cc:1620] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:56:56.066581 24448 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 7b41045eecf144bc9acdd963622c4c30 (table rebalancer_dynamic_table [id=70b90d9d6fc34e6fa16ca554a9cb411a]). Expected version 1 got 0
I20240501 19:56:56.069754 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 0ed4b95e555a46c5976ca9e084737ea1 (table rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]) successfully deleted
I20240501 19:56:56.069880 24448 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 7b41045eecf144bc9acdd963622c4c30 (table rebalancer_dynamic_table [id=70b90d9d6fc34e6fa16ca554a9cb411a]). Expected version 1 got 0
I20240501 19:56:56.071136 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=70b90d9d6fc34e6fa16ca554a9cb411a] alter complete (version 1)
I20240501 19:56:56.071563 31159 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:56.071875 31159 tablet.cc:1620] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:56.070387 31404 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:56.072134 31404 tablet.cc:1620] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:56.111084 31297 raft_consensus.cc:482] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:56.111207 31297 raft_consensus.cc:504] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.111582 31297 leader_election.cc:288] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:56.111876 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8020390483c4fd98265cbe7d8012435" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:56.111964 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8020390483c4fd98265cbe7d8012435" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:56.112010 27278 raft_consensus.cc:2454] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:56.112080 27162 raft_consensus.cc:2454] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:56:56.112265 24793 leader_election.cc:302] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:56.112411 31297 raft_consensus.cc:2781] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:56.112468 31297 raft_consensus.cc:482] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:56.112509 31297 raft_consensus.cc:3037] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.113535 31297 raft_consensus.cc:504] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.113960 31297 leader_election.cc:288] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:56.115813 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8020390483c4fd98265cbe7d8012435" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:56.115824 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8020390483c4fd98265cbe7d8012435" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:56.115936 27278 raft_consensus.cc:3037] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.115936 27162 raft_consensus.cc:3037] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.116847 27162 raft_consensus.cc:2454] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:56.116961 27278 raft_consensus.cc:2454] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:56:56.117211 24793 leader_election.cc:302] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:56.117357 31297 raft_consensus.cc:2781] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:56.117453 31297 raft_consensus.cc:686] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:56:56.117585 31297 consensus_queue.cc:237] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.122432 24454 catalog_manager.cc:5484] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:56.176262 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:56.182435 27143 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:56.182735 27012 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:56.186291 31403 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:56.186684 31403 tablet.cc:1620] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:56:56.189142 31404 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:56.189443 31404 tablet.cc:1620] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:56.189960 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:56.190043 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:56.200716 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=70b90d9d6fc34e6fa16ca554a9cb411a] alter complete (version 2)
I20240501 19:56:56.206092 31159 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:56.206405 31159 tablet.cc:1620] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:56.216423 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:56.220461 27273 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:56.227695 27404 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:56.232333 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:56.278715 24857 consensus_queue.cc:237] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.280190 27425 raft_consensus.cc:1261] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:56.280310 24725 raft_consensus.cc:1261] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:56.280699 30151 consensus_queue.cc:1035] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:56.280881 31297 consensus_queue.cc:1035] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:56.282717 30151 raft_consensus.cc:2932] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:56.283104 27425 raft_consensus.cc:2932] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:56.283715 24725 raft_consensus.cc:2932] T 92dd4bc514f146bb89083fe8a9315308 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:56.286453 24453 catalog_manager.cc:5484] T 92dd4bc514f146bb89083fe8a9315308 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:56.289705 24967 tablet_service.cc:1514] Processing DeleteTablet for tablet 92dd4bc514f146bb89083fe8a9315308 with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:56.290979 31477 tablet_replica.cc:330] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:56.291157 31477 raft_consensus.cc:2229] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.291303 31477 raft_consensus.cc:2258] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.291728 31477 ts_tablet_manager.cc:1875] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:56.293453 31477 ts_tablet_manager.cc:1888] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:56.293526 31477 log.cc:1196] T 92dd4bc514f146bb89083fe8a9315308 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/92dd4bc514f146bb89083fe8a9315308
I20240501 19:56:56.293975 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 92dd4bc514f146bb89083fe8a9315308 (table rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]) successfully deleted
I20240501 19:56:56.304105 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:56.308883 31403 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:56.309675 31404 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:56.309672 31159 tablet.cc:1722] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:56.314394 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=70b90d9d6fc34e6fa16ca554a9cb411a] alter complete (version 3)
I20240501 19:56:56.386771 27143 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:56.388442 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:56.392247 27404 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:56.392467 27012 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:56.392632 27273 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:56.393198 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:56.394191 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:56.416371 24448 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:56.416545 24448 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:56.418689 31374 consensus_queue.cc:1035] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.419275 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 7b41045eecf144bc9acdd963622c4c30
I20240501 19:56:56.419979 31374 consensus_queue.cc:1035] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.420814 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b41045eecf144bc9acdd963622c4c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:56.421293 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b41045eecf144bc9acdd963622c4c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:56 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:56.421802 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b41045eecf144bc9acdd963622c4c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:56 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:56.422689 31147 tablet_replica.cc:330] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:56.422848 31147 raft_consensus.cc:2229] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:56.423367 31147 raft_consensus.cc:2258] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.424374 31147 ts_tablet_manager.cc:1875] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:56.426595 31147 ts_tablet_manager.cc:1888] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:56.426688 31147 log.cc:1196] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/7b41045eecf144bc9acdd963622c4c30
I20240501 19:56:56.426888 31147 ts_tablet_manager.cc:1909] T 7b41045eecf144bc9acdd963622c4c30 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:56.428575 31527 tablet_replica.cc:330] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:56.428725 31527 raft_consensus.cc:2229] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.428843 31527 raft_consensus.cc:2258] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.429275 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 7b41045eecf144bc9acdd963622c4c30 (table rebalancer_dynamic_table_new_name [id=70b90d9d6fc34e6fa16ca554a9cb411a]) successfully deleted
I20240501 19:56:56.429483 31527 ts_tablet_manager.cc:1875] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:56.439742 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:56.440716 31527 ts_tablet_manager.cc:1888] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:56.443080 31527 log.cc:1196] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/7b41045eecf144bc9acdd963622c4c30
I20240501 19:56:56.443449 31527 ts_tablet_manager.cc:1909] T 7b41045eecf144bc9acdd963622c4c30 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:56.444550 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 7b41045eecf144bc9acdd963622c4c30 (table rebalancer_dynamic_table_new_name [id=70b90d9d6fc34e6fa16ca554a9cb411a]) successfully deleted
I20240501 19:56:56.455565 31528 tablet_replica.cc:330] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:56.455807 31528 raft_consensus.cc:2229] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.455960 31528 raft_consensus.cc:2258] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.457084 31528 ts_tablet_manager.cc:1875] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:56.465118 31528 ts_tablet_manager.cc:1888] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:56.465261 31528 log.cc:1196] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/7b41045eecf144bc9acdd963622c4c30
I20240501 19:56:56.465670 31528 ts_tablet_manager.cc:1909] T 7b41045eecf144bc9acdd963622c4c30 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:56.466421 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 7b41045eecf144bc9acdd963622c4c30 (table rebalancer_dynamic_table_new_name [id=70b90d9d6fc34e6fa16ca554a9cb411a]) successfully deleted
I20240501 19:56:56.488778 24857 consensus_queue.cc:237] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.489913 27162 raft_consensus.cc:1261] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:56.489948 24986 raft_consensus.cc:1261] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:56.490280 30151 consensus_queue.cc:1035] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:56.490442 31297 consensus_queue.cc:1035] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:56.492161 31550 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:56.492507 27162 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:56.492560 24986 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:56.495225 24448 catalog_manager.cc:5484] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:56.498121 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b692b5b96534e79a32a8b06b92f32d6 with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:56.498399 31528 tablet_replica.cc:330] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:56.498553 31528 raft_consensus.cc:2229] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.498696 31528 raft_consensus.cc:2258] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.499261 31528 ts_tablet_manager.cc:1875] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:56.501493 31528 ts_tablet_manager.cc:1888] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:56.501997 31528 log.cc:1196] T 3b692b5b96534e79a32a8b06b92f32d6 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/3b692b5b96534e79a32a8b06b92f32d6
I20240501 19:56:56.502614 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 3b692b5b96534e79a32a8b06b92f32d6 (table rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]) successfully deleted
I20240501 19:56:56.521932 24452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:56.527881 27273 tablet_service.cc:1467] Processing CreateTablet for tablet e2ba732ba6f24aff85db87efe2cbe707 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2cf36e3a7e4f57bc9a9298f059ca99]), partition=
I20240501 19:56:56.528020 24837 tablet_service.cc:1467] Processing CreateTablet for tablet e2ba732ba6f24aff85db87efe2cbe707 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2cf36e3a7e4f57bc9a9298f059ca99]), partition=
I20240501 19:56:56.528169 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ba732ba6f24aff85db87efe2cbe707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.528281 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ba732ba6f24aff85db87efe2cbe707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.528318 27012 tablet_service.cc:1467] Processing CreateTablet for tablet e2ba732ba6f24aff85db87efe2cbe707 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2cf36e3a7e4f57bc9a9298f059ca99]), partition=
I20240501 19:56:56.528533 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ba732ba6f24aff85db87efe2cbe707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.530076 31566 tablet_bootstrap.cc:492] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:56.530977 31569 tablet_bootstrap.cc:492] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:56.531015 31568 tablet_bootstrap.cc:492] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:56.532104 31566 tablet_bootstrap.cc:654] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.532104 31569 tablet_bootstrap.cc:654] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.533016 31568 tablet_bootstrap.cc:654] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.533520 31569 tablet_bootstrap.cc:492] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:56.533654 31569 ts_tablet_manager.cc:1367] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:56.534175 31569 raft_consensus.cc:348] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.534245 31568 tablet_bootstrap.cc:492] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:56.534323 31568 ts_tablet_manager.cc:1367] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:56.534440 31569 raft_consensus.cc:374] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.534521 31569 raft_consensus.cc:729] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.534734 31569 consensus_queue.cc:260] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.534730 31568 raft_consensus.cc:348] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.534829 31568 raft_consensus.cc:374] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.534859 31568 raft_consensus.cc:729] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.535006 31568 consensus_queue.cc:260] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.535429 31568 ts_tablet_manager.cc:1398] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:56.535830 31569 ts_tablet_manager.cc:1398] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:56.537142 31566 tablet_bootstrap.cc:492] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:56.537277 31566 ts_tablet_manager.cc:1367] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:56.537645 31566 raft_consensus.cc:348] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.537742 31566 raft_consensus.cc:374] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.537778 31566 raft_consensus.cc:729] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.537950 31566 consensus_queue.cc:260] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.538352 31566 ts_tablet_manager.cc:1398] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:56.558436 31142 consensus_queue.cc:1035] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.564546 29464 consensus_queue.cc:1035] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.631997 31550 consensus_queue.cc:1035] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.634785 31550 consensus_queue.cc:1035] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.644165 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:56.650278 27143 tablet_service.cc:1467] Processing CreateTablet for tablet b4b629378083406b932bfbd5407cac3a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.650638 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b629378083406b932bfbd5407cac3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.650969 27010 tablet_service.cc:1467] Processing CreateTablet for tablet abb4f8ef27a54eea89241362c525553b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.651324 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb4f8ef27a54eea89241362c525553b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.651309 24705 tablet_service.cc:1467] Processing CreateTablet for tablet b4b629378083406b932bfbd5407cac3a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.651557 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b629378083406b932bfbd5407cac3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.651824 27267 tablet_service.cc:1467] Processing CreateTablet for tablet abb4f8ef27a54eea89241362c525553b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.652060 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb4f8ef27a54eea89241362c525553b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.653476 31569 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:56.653623 27273 tablet_service.cc:1467] Processing CreateTablet for tablet c53ceef9ab1d4ec593f1dfe7bd8695e5 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.653844 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c53ceef9ab1d4ec593f1dfe7bd8695e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.650974 27012 tablet_service.cc:1467] Processing CreateTablet for tablet c53ceef9ab1d4ec593f1dfe7bd8695e5 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.654049 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c53ceef9ab1d4ec593f1dfe7bd8695e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.654395 27404 tablet_service.cc:1467] Processing CreateTablet for tablet b4b629378083406b932bfbd5407cac3a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.654512 31569 tablet_bootstrap.cc:654] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.654647 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b629378083406b932bfbd5407cac3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.655897 27406 tablet_service.cc:1467] Processing CreateTablet for tablet abb4f8ef27a54eea89241362c525553b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.656105 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb4f8ef27a54eea89241362c525553b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.656984 31606 tablet_bootstrap.cc:492] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:56.656998 31566 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:56.657930 31566 tablet_bootstrap.cc:654] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.658155 31604 tablet_bootstrap.cc:492] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:56.658470 31606 tablet_bootstrap.cc:654] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.659188 31604 tablet_bootstrap.cc:654] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.659726 31566 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:56.659817 31566 ts_tablet_manager.cc:1367] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:56.660109 27143 tablet_service.cc:1467] Processing CreateTablet for tablet c53ceef9ab1d4ec593f1dfe7bd8695e5 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b12e4f4f7fe94a84bc10f1ccdaf184f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:56.660190 31566 raft_consensus.cc:348] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.660305 31566 raft_consensus.cc:374] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.660331 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c53ceef9ab1d4ec593f1dfe7bd8695e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.660343 31566 raft_consensus.cc:729] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.660463 31566 consensus_queue.cc:260] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.660720 31566 ts_tablet_manager.cc:1398] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:56.660892 31566 tablet_bootstrap.cc:492] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:56.661015 31604 tablet_bootstrap.cc:492] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:56.661080 31604 ts_tablet_manager.cc:1367] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:56.661496 31604 raft_consensus.cc:348] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.661586 31604 raft_consensus.cc:374] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.661625 31604 raft_consensus.cc:729] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.661721 31566 tablet_bootstrap.cc:654] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.661865 31604 consensus_queue.cc:260] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.662755 31604 ts_tablet_manager.cc:1398] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:56.663049 31604 tablet_bootstrap.cc:492] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:56.663309 31606 tablet_bootstrap.cc:492] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:56.663456 31606 ts_tablet_manager.cc:1367] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 19:56:56.663928 31606 raft_consensus.cc:348] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.664027 31604 tablet_bootstrap.cc:654] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.664099 31606 raft_consensus.cc:374] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.664182 31606 raft_consensus.cc:729] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.664412 31606 consensus_queue.cc:260] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.664903 31606 ts_tablet_manager.cc:1398] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:56.665127 31606 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:56.666090 31606 tablet_bootstrap.cc:654] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.667011 31604 tablet_bootstrap.cc:492] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:56.667171 31604 ts_tablet_manager.cc:1367] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:56.667585 31607 tablet_bootstrap.cc:492] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:56.667917 31604 raft_consensus.cc:348] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.668195 31604 raft_consensus.cc:374] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.668329 31604 raft_consensus.cc:729] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.668532 31604 consensus_queue.cc:260] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.668735 31569 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:56.668821 31569 ts_tablet_manager.cc:1367] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240501 19:56:56.668920 31604 ts_tablet_manager.cc:1398] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:56.669273 31569 raft_consensus.cc:348] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.669373 31569 raft_consensus.cc:374] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.669394 31569 raft_consensus.cc:729] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.669505 31606 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:56.669492 31569 consensus_queue.cc:260] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.669560 31606 ts_tablet_manager.cc:1367] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:56.669736 31569 ts_tablet_manager.cc:1398] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:56.669873 31569 tablet_bootstrap.cc:492] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:56.669893 31606 raft_consensus.cc:348] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.669986 31606 raft_consensus.cc:374] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.670007 31606 raft_consensus.cc:729] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.670111 31606 consensus_queue.cc:260] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.670424 31606 ts_tablet_manager.cc:1398] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:56.670732 31569 tablet_bootstrap.cc:654] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.672302 30935 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:56.672385 30935 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.672719 31569 tablet_bootstrap.cc:492] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:56.672741 30935 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:56.672796 31569 ts_tablet_manager.cc:1367] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:56.673192 31569 raft_consensus.cc:348] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.673206 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b629378083406b932bfbd5407cac3a" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:56.673290 31569 raft_consensus.cc:374] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.673322 31569 raft_consensus.cc:729] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.673446 31569 consensus_queue.cc:260] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
W20240501 19:56:56.673547 27359 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:56.673722 31569 ts_tablet_manager.cc:1398] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:56.674139 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b629378083406b932bfbd5407cac3a" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:56.674237 27162 raft_consensus.cc:2454] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:56.674481 27361 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters: 9081292597be4a72a42b216fddcd7971
I20240501 19:56:56.674965 30935 raft_consensus.cc:2781] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:56.675014 30935 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:56.675083 30935 raft_consensus.cc:3037] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.676070 30935 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.676380 30935 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:56.676585 31607 tablet_bootstrap.cc:654] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:56.676730 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b629378083406b932bfbd5407cac3a" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
W20240501 19:56:56.677214 27359 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:56.677548 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b629378083406b932bfbd5407cac3a" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:56.677649 27162 raft_consensus.cc:3037] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.678606 31607 tablet_bootstrap.cc:492] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:56.678598 27162 raft_consensus.cc:2454] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:56.678674 31566 tablet_bootstrap.cc:492] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:56.678699 31607 ts_tablet_manager.cc:1367] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:56.678766 31566 ts_tablet_manager.cc:1367] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240501 19:56:56.678853 27361 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters: 9081292597be4a72a42b216fddcd7971
I20240501 19:56:56.679067 30935 raft_consensus.cc:2781] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:56.679236 30935 raft_consensus.cc:686] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:56.679211 31607 raft_consensus.cc:348] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.679250 31566 raft_consensus.cc:348] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.679352 31566 raft_consensus.cc:374] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.679355 30935 consensus_queue.cc:237] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.679386 31566 raft_consensus.cc:729] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.679306 31607 raft_consensus.cc:374] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:56.679489 31607 raft_consensus.cc:729] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:56.679519 31566 consensus_queue.cc:260] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.679632 31607 consensus_queue.cc:260] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:56.679819 31566 ts_tablet_manager.cc:1398] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:56.679999 31607 ts_tablet_manager.cc:1398] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:56.682102 24448 catalog_manager.cc:5484] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:56.702450 29464 raft_consensus.cc:482] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:56.702566 29464 raft_consensus.cc:504] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.702919 29464 leader_election.cc:288] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:56.703244 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ba732ba6f24aff85db87efe2cbe707" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:56.703317 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ba732ba6f24aff85db87efe2cbe707" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:56.703397 24857 raft_consensus.cc:2454] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:56.703460 27032 raft_consensus.cc:2454] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:56.703685 27228 leader_election.cc:302] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:56.703836 29464 raft_consensus.cc:2781] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:56.703893 29464 raft_consensus.cc:482] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:56.703934 29464 raft_consensus.cc:3037] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.704744 29464 raft_consensus.cc:504] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.705058 29464 leader_election.cc:288] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:56.705322 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ba732ba6f24aff85db87efe2cbe707" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:56.705366 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ba732ba6f24aff85db87efe2cbe707" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:56.705427 24857 raft_consensus.cc:3037] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.705471 27032 raft_consensus.cc:3037] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.706429 24857 raft_consensus.cc:2454] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:56.706439 27032 raft_consensus.cc:2454] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:56.706698 27228 leader_election.cc:302] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:56.706874 29464 raft_consensus.cc:2781] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:56.706955 29464 raft_consensus.cc:686] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:56.707118 29464 consensus_queue.cc:237] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:56.710659 24453 catalog_manager.cc:5484] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:56.727566 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:56.732192 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:56.736979 27012 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:56.737136 27143 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:56.742390 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:56.751225 27273 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:56.763212 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:56.764873 27406 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:56.823155 31374 raft_consensus.cc:482] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:56.823294 31374 raft_consensus.cc:504] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.823627 31374 leader_election.cc:288] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:56.824040 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c53ceef9ab1d4ec593f1dfe7bd8695e5" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:56.824201 27162 raft_consensus.cc:2454] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:56.824188 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c53ceef9ab1d4ec593f1dfe7bd8695e5" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:56.824311 27276 raft_consensus.cc:2454] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:56.824476 26967 leader_election.cc:302] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:56:56.824640 31374 raft_consensus.cc:2781] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:56.824698 31374 raft_consensus.cc:482] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:56.824745 31374 raft_consensus.cc:3037] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.825621 31374 raft_consensus.cc:504] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.825922 31374 leader_election.cc:288] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:56.826244 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c53ceef9ab1d4ec593f1dfe7bd8695e5" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:56.826282 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c53ceef9ab1d4ec593f1dfe7bd8695e5" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:56.826359 27276 raft_consensus.cc:3037] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.826390 27162 raft_consensus.cc:3037] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.827206 27276 raft_consensus.cc:2454] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:56.827337 27162 raft_consensus.cc:2454] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:56.827472 26967 leader_election.cc:302] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:56.827625 31374 raft_consensus.cc:2781] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:56.827781 31374 raft_consensus.cc:686] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:56.827960 31374 consensus_queue.cc:237] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:56.829207 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:56.830567 24448 catalog_manager.cc:5484] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:56.833665 27032 raft_consensus.cc:1261] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:56.833922 24857 raft_consensus.cc:1261] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:56.834218 31142 consensus_queue.cc:1035] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.834735 31142 consensus_queue.cc:1035] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:56.836831 24448 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet e2ba732ba6f24aff85db87efe2cbe707 (table rebalancer_dynamic_table [id=eb2cf36e3a7e4f57bc9a9298f059ca99]). Expected version 1 got 0
I20240501 19:56:56.838873 31645 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:56.839200 24453 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet e2ba732ba6f24aff85db87efe2cbe707 (table rebalancer_dynamic_table [id=eb2cf36e3a7e4f57bc9a9298f059ca99]). Expected version 1 got 0
I20240501 19:56:56.839416 31645 tablet.cc:1620] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:56:56.840281 31644 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:56.840636 31644 tablet.cc:1620] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:56.841156 31648 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:56.841338 24448 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet e2ba732ba6f24aff85db87efe2cbe707 (table rebalancer_dynamic_table [id=eb2cf36e3a7e4f57bc9a9298f059ca99]). Expected version 1 got 0
I20240501 19:56:56.841456 31648 tablet.cc:1620] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:56.843997 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=eb2cf36e3a7e4f57bc9a9298f059ca99] alter complete (version 1)
I20240501 19:56:56.847867 24725 consensus_queue.cc:237] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.848989 24857 raft_consensus.cc:1261] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:56.848996 27276 raft_consensus.cc:1261] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:56.849314 30378 consensus_queue.cc:1035] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:56.849468 31099 consensus_queue.cc:1035] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:56.851130 31655 raft_consensus.cc:2932] T 164460e6fbc44fb1a9a12d52fea9254b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.851416 27276 raft_consensus.cc:2932] T 164460e6fbc44fb1a9a12d52fea9254b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.851447 24857 raft_consensus.cc:2932] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.853983 24453 catalog_manager.cc:5484] T 164460e6fbc44fb1a9a12d52fea9254b P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.857892 24967 tablet_service.cc:1514] Processing DeleteTablet for tablet 164460e6fbc44fb1a9a12d52fea9254b with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:56.858201 31657 tablet_replica.cc:330] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:56.858373 31657 raft_consensus.cc:2229] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.858543 31657 raft_consensus.cc:2258] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.859138 31657 ts_tablet_manager.cc:1875] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:56.861377 31657 ts_tablet_manager.cc:1888] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:56.861450 31657 log.cc:1196] T 164460e6fbc44fb1a9a12d52fea9254b P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/164460e6fbc44fb1a9a12d52fea9254b
I20240501 19:56:56.861920 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 164460e6fbc44fb1a9a12d52fea9254b (table rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]) successfully deleted
I20240501 19:56:56.904024 30967 raft_consensus.cc:482] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:56.904147 30967 raft_consensus.cc:504] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.904572 30967 leader_election.cc:288] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:56.904904 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abb4f8ef27a54eea89241362c525553b" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:56.904938 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abb4f8ef27a54eea89241362c525553b" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:56.905057 27424 raft_consensus.cc:2454] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:56.905066 27276 raft_consensus.cc:2454] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:56.905319 26967 leader_election.cc:302] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:56.905503 30967 raft_consensus.cc:2781] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:56.905566 30967 raft_consensus.cc:482] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:56.905606 30967 raft_consensus.cc:3037] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.906797 30967 raft_consensus.cc:504] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.907155 30967 leader_election.cc:288] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:56.907722 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abb4f8ef27a54eea89241362c525553b" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:56.907835 27276 raft_consensus.cc:3037] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.908843 27276 raft_consensus.cc:2454] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:56.909240 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abb4f8ef27a54eea89241362c525553b" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:56.909358 27424 raft_consensus.cc:3037] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:56.910377 27424 raft_consensus.cc:2454] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:56.910737 26967 leader_election.cc:302] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:56.910965 30967 raft_consensus.cc:2781] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:56.911063 30967 raft_consensus.cc:686] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:56.911211 30967 consensus_queue.cc:237] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:56.913360 24448 catalog_manager.cc:5484] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:56.948422 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:56.956388 31644 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:56.956759 31644 tablet.cc:1620] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:56.957067 27012 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:56.957937 31645 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:56.958200 31645 tablet.cc:1620] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:56:56.962425 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:56.963929 31648 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:56.964229 31648 tablet.cc:1620] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:56.965627 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:56.969137 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:56.969437 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:56.969776 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=eb2cf36e3a7e4f57bc9a9298f059ca99] alter complete (version 2)
I20240501 19:56:56.983948 27406 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:56.985757 27273 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:56.987354 27143 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:57.051641 24725 consensus_queue.cc:237] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:56:57.052894 24594 raft_consensus.cc:1261] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:57.052908 24856 raft_consensus.cc:1261] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:57.053226 31099 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:57.053311 31099 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:57.053321 31656 consensus_queue.cc:1035] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:57.053522 31655 consensus_queue.cc:1035] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:57.053625 31099 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:57.054019 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b629378083406b932bfbd5407cac3a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:57.054109 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b629378083406b932bfbd5407cac3a" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:57.054286 27162 raft_consensus.cc:2379] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9081292597be4a72a42b216fddcd7971 in current term 1: Already voted for candidate 4a6e54e86a9949e5970b42e8e32e7561 in this term.
I20240501 19:56:57.054765 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:57.054932 31655 raft_consensus.cc:3037] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.055363 31099 raft_consensus.cc:2932] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:57.055783 24594 raft_consensus.cc:2932] T 9a6b9bbaf2a0470482f3c5ff98322b90 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:57.056082 31655 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:57.057674 24856 raft_consensus.cc:2932] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:57.058595 24448 catalog_manager.cc:5484] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:56:57.061902 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6b9bbaf2a0470482f3c5ff98322b90 with delete_type TABLET_DATA_TOMBSTONED (TS 3b5968cebd0b4294898fdd89b7a5b3a1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:57.062292 31733 tablet_replica.cc:330] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:57.062448 31733 raft_consensus.cc:2229] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:57.062628 31733 raft_consensus.cc:2258] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.063138 31733 ts_tablet_manager.cc:1875] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:57.065636 31733 ts_tablet_manager.cc:1888] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:56:57.065807 31733 log.cc:1196] T 9a6b9bbaf2a0470482f3c5ff98322b90 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/9a6b9bbaf2a0470482f3c5ff98322b90
I20240501 19:56:57.066383 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 9a6b9bbaf2a0470482f3c5ff98322b90 (table rebalancer_extra_table_8 [id=9b5757f7de69488f9ff41d5f89192b01]) successfully deleted
I20240501 19:56:57.085167 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:57.092123 31644 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:57.092526 31648 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:57.093289 31645 tablet.cc:1722] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:57.097582 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=eb2cf36e3a7e4f57bc9a9298f059ca99] alter complete (version 3)
I20240501 19:56:57.157302 27273 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:57.158593 27406 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:57.158849 27143 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:57.163872 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.166306 27012 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:57.183084 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:57.194075 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:57.203922 24453 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:57.204129 24453 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:57.206297 24453 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet e2ba732ba6f24aff85db87efe2cbe707
I20240501 19:56:57.207273 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ba732ba6f24aff85db87efe2cbe707 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:57 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:57.207413 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ba732ba6f24aff85db87efe2cbe707 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:57 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:57.207481 31733 tablet_replica.cc:330] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:57.207626 31733 raft_consensus.cc:2229] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:57.207754 31733 raft_consensus.cc:2258] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.208453 31733 ts_tablet_manager.cc:1875] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:57.210156 27273 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ba732ba6f24aff85db87efe2cbe707 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:57 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:56:57.210547 31733 ts_tablet_manager.cc:1888] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:57.210614 31733 log.cc:1196] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/e2ba732ba6f24aff85db87efe2cbe707
I20240501 19:56:57.211949 31733 ts_tablet_manager.cc:1909] T e2ba732ba6f24aff85db87efe2cbe707 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:57.212741 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet e2ba732ba6f24aff85db87efe2cbe707 (table rebalancer_dynamic_table_new_name [id=eb2cf36e3a7e4f57bc9a9298f059ca99]) successfully deleted
I20240501 19:56:57.218322 31783 tablet_replica.cc:330] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:57.218565 31784 tablet_replica.cc:330] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:56:57.218765 31784 raft_consensus.cc:2229] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:57.218926 31783 raft_consensus.cc:2229] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:57.219080 31783 raft_consensus.cc:2258] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.219375 31784 raft_consensus.cc:2258] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.222573 31784 ts_tablet_manager.cc:1875] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:57.223440 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.223066 31783 ts_tablet_manager.cc:1875] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:57.232934 31783 ts_tablet_manager.cc:1888] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:57.233026 31783 log.cc:1196] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/e2ba732ba6f24aff85db87efe2cbe707
I20240501 19:56:57.233314 31783 ts_tablet_manager.cc:1909] T e2ba732ba6f24aff85db87efe2cbe707 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:57.233973 31784 ts_tablet_manager.cc:1888] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:57.234048 31784 log.cc:1196] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/e2ba732ba6f24aff85db87efe2cbe707
I20240501 19:56:57.234314 31784 ts_tablet_manager.cc:1909] T e2ba732ba6f24aff85db87efe2cbe707 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:56:57.235208 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet e2ba732ba6f24aff85db87efe2cbe707 (table rebalancer_dynamic_table_new_name [id=eb2cf36e3a7e4f57bc9a9298f059ca99]) successfully deleted
I20240501 19:56:57.237113 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet e2ba732ba6f24aff85db87efe2cbe707 (table rebalancer_dynamic_table_new_name [id=eb2cf36e3a7e4f57bc9a9298f059ca99]) successfully deleted
I20240501 19:56:57.267416 24987 consensus_queue.cc:237] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:57.268685 27032 raft_consensus.cc:1261] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:57.268685 24856 raft_consensus.cc:1261] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:57.269115 31630 consensus_queue.cc:1035] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:57.269301 31280 consensus_queue.cc:1035] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:57.271340 31630 raft_consensus.cc:2932] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:57.271782 27032 raft_consensus.cc:2932] T 23ae3af247c1456e8b75640c25906faf P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:57.271786 24856 raft_consensus.cc:2932] T 23ae3af247c1456e8b75640c25906faf P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:57.274824 24453 catalog_manager.cc:5484] T 23ae3af247c1456e8b75640c25906faf P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:57.276626 30935 consensus_queue.cc:1035] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:57.279157 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ae3af247c1456e8b75640c25906faf with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:57.280951 30935 consensus_queue.cc:1035] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:57.281636 31811 tablet_replica.cc:330] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:57.281832 31811 raft_consensus.cc:2229] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:57.282052 31811 raft_consensus.cc:2258] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.283253 31811 ts_tablet_manager.cc:1875] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:57.286128 31811 ts_tablet_manager.cc:1888] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:57.286247 31811 log.cc:1196] T 23ae3af247c1456e8b75640c25906faf P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/23ae3af247c1456e8b75640c25906faf
I20240501 19:56:57.286942 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 23ae3af247c1456e8b75640c25906faf (table rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]) successfully deleted
I20240501 19:56:57.300709 30967 consensus_queue.cc:1035] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:57.306684 30967 consensus_queue.cc:1035] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:57.311362 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:57.311614 31374 consensus_queue.cc:1035] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:57.316196 30967 consensus_queue.cc:1035] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:57.319070 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 8d527149db554c0c8c2472d1374c0d1f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc]), partition=
I20240501 19:56:57.319278 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 8d527149db554c0c8c2472d1374c0d1f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc]), partition=
I20240501 19:56:57.319336 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d527149db554c0c8c2472d1374c0d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.319520 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d527149db554c0c8c2472d1374c0d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.321303 31844 tablet_bootstrap.cc:492] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:57.321642 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 8d527149db554c0c8c2472d1374c0d1f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc]), partition=
I20240501 19:56:57.321890 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d527149db554c0c8c2472d1374c0d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.322351 31844 tablet_bootstrap.cc:654] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.323251 31845 tablet_bootstrap.cc:492] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:57.323447 31844 tablet_bootstrap.cc:492] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:57.323529 31844 ts_tablet_manager.cc:1367] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:57.323951 31844 raft_consensus.cc:348] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.324051 31844 raft_consensus.cc:374] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.324085 31845 tablet_bootstrap.cc:654] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.324100 31844 raft_consensus.cc:729] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.324219 31844 consensus_queue.cc:260] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.324513 31844 ts_tablet_manager.cc:1398] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:57.325937 31847 tablet_bootstrap.cc:492] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:57.327531 31847 tablet_bootstrap.cc:654] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.328701 31847 tablet_bootstrap.cc:492] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:57.328838 31847 ts_tablet_manager.cc:1367] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:57.329056 31845 tablet_bootstrap.cc:492] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:57.329217 31845 ts_tablet_manager.cc:1367] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:57.329417 31847 raft_consensus.cc:348] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.329593 31847 raft_consensus.cc:374] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.329675 31847 raft_consensus.cc:729] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.329880 31847 consensus_queue.cc:260] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.329871 31845 raft_consensus.cc:348] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.329998 31845 raft_consensus.cc:374] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.330034 31845 raft_consensus.cc:729] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.330286 31847 ts_tablet_manager.cc:1398] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:57.330250 31845 consensus_queue.cc:260] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.330776 31845 ts_tablet_manager.cc:1398] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:57.350971 24725 consensus_queue.cc:237] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:57.352581 24987 raft_consensus.cc:1261] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:57.352617 24660 consensus_peers.cc:485] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f1f7372e5ce42b284cf1179c72f578d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:57.352623 24856 raft_consensus.cc:1261] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:57.352995 31656 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:57.353174 30378 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:57.355178 31656 raft_consensus.cc:2932] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:57.355636 24856 raft_consensus.cc:2932] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:57.357326 24987 raft_consensus.cc:2932] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:57.358412 24451 catalog_manager.cc:5484] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:57.381166 24594 consensus_queue.cc:237] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:57.382712 24856 raft_consensus.cc:1261] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:57.382824 24987 raft_consensus.cc:1261] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:57.383064 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e30faedce3742e0ba83de21059613d1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:57.383208 31879 consensus_queue.cc:1035] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:57.383396 31091 consensus_queue.cc:1035] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:57.385571 31091 raft_consensus.cc:2932] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:57.385954 24987 raft_consensus.cc:2932] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:57.387612 24856 raft_consensus.cc:2932] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:57.388782 24453 catalog_manager.cc:5484] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:57.417241 27163 consensus_queue.cc:237] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:57.418781 27276 raft_consensus.cc:1261] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:57.418781 27425 raft_consensus.cc:1261] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:57.418979 27097 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a4c18dfded64901bb4645af3dbe2dbc. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:57.419298 31899 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:57.419505 31900 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:57.421859 31900 raft_consensus.cc:2932] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:57.422292 27425 raft_consensus.cc:2932] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:57.424048 27276 raft_consensus.cc:2932] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:57.427107 24446 catalog_manager.cc:5484] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:57.447521 27163 consensus_queue.cc:237] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:57.448921 27032 raft_consensus.cc:1261] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:57.449113 27097 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a44d1713369478191bfc30457bf61c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:57.449146 27276 raft_consensus.cc:1261] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:57.449316 31900 consensus_queue.cc:1035] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:57.449527 31406 consensus_queue.cc:1035] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:57.451706 31406 raft_consensus.cc:2932] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:57.451946 31900 raft_consensus.cc:482] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:57.452023 31900 raft_consensus.cc:504] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.452430 31900 leader_election.cc:288] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:57.452809 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d527149db554c0c8c2472d1374c0d1f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:57.452966 27425 raft_consensus.cc:2454] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:57.453290 27098 leader_election.cc:302] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:57.453441 31900 raft_consensus.cc:2781] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:57.453497 31900 raft_consensus.cc:482] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:57.453459 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d527149db554c0c8c2472d1374c0d1f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:57.453552 31900 raft_consensus.cc:3037] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.453605 24594 raft_consensus.cc:2454] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:57.453796 27276 raft_consensus.cc:2932] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:57.453930 27032 raft_consensus.cc:2932] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:57.454389 31900 raft_consensus.cc:504] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.454725 31900 leader_election.cc:288] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:57.455637 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d527149db554c0c8c2472d1374c0d1f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:57.455763 24594 raft_consensus.cc:3037] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.456487 24453 catalog_manager.cc:5484] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:57.456790 24594 raft_consensus.cc:2454] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:57.457217 27096 leader_election.cc:302] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:57.457387 31900 raft_consensus.cc:2781] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:57.457475 31900 raft_consensus.cc:686] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:57.457594 31900 consensus_queue.cc:237] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.460390 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d527149db554c0c8c2472d1374c0d1f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:57.460520 27425 raft_consensus.cc:3037] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.460672 24453 catalog_manager.cc:5484] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:57.461566 27425 raft_consensus.cc:2454] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:57.464615 24445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:57.470897 27012 tablet_service.cc:1467] Processing CreateTablet for tablet a1e40a2230d149cd91d389c9f2b451f6 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.471248 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e40a2230d149cd91d389c9f2b451f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.471685 27273 tablet_service.cc:1467] Processing CreateTablet for tablet a1e40a2230d149cd91d389c9f2b451f6 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.471930 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e40a2230d149cd91d389c9f2b451f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.472043 27406 tablet_service.cc:1467] Processing CreateTablet for tablet a1e40a2230d149cd91d389c9f2b451f6 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.472093 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 85c67171da5e4b8bbfce0398461ae902 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.472291 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e40a2230d149cd91d389c9f2b451f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.472324 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c67171da5e4b8bbfce0398461ae902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.474248 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 85c67171da5e4b8bbfce0398461ae902 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.474431 31847 tablet_bootstrap.cc:492] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:57.474467 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c67171da5e4b8bbfce0398461ae902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.474445 27404 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad3e4102cd14025b897ac7da10efe9e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.474722 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad3e4102cd14025b897ac7da10efe9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.474817 31933 tablet_bootstrap.cc:492] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:57.475425 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad3e4102cd14025b897ac7da10efe9e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.475751 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad3e4102cd14025b897ac7da10efe9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.475767 31933 tablet_bootstrap.cc:654] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.476876 31933 tablet_bootstrap.cc:492] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:57.476953 31933 ts_tablet_manager.cc:1367] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:57.477402 31933 raft_consensus.cc:348] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.477499 31933 raft_consensus.cc:374] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.477530 31933 raft_consensus.cc:729] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.477641 31933 consensus_queue.cc:260] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.477806 31847 tablet_bootstrap.cc:654] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.478001 31933 ts_tablet_manager.cc:1398] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:57.478152 31933 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:57.479053 31933 tablet_bootstrap.cc:654] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.479808 31844 tablet_bootstrap.cc:492] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:57.480304 27010 tablet_service.cc:1467] Processing CreateTablet for tablet 85c67171da5e4b8bbfce0398461ae902 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.480542 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c67171da5e4b8bbfce0398461ae902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.480737 31844 tablet_bootstrap.cc:654] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.481992 31932 tablet_bootstrap.cc:492] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:57.482857 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad3e4102cd14025b897ac7da10efe9e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:57.482944 31932 tablet_bootstrap.cc:654] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.483206 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad3e4102cd14025b897ac7da10efe9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.484122 31932 tablet_bootstrap.cc:492] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:57.484205 31932 ts_tablet_manager.cc:1367] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:57.484666 31932 raft_consensus.cc:348] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.484761 31932 raft_consensus.cc:374] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.484791 31932 raft_consensus.cc:729] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.484897 31932 consensus_queue.cc:260] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.485157 31933 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:57.485181 31932 ts_tablet_manager.cc:1398] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:57.485234 31933 ts_tablet_manager.cc:1367] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:56:57.485378 31932 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:57.485610 31933 raft_consensus.cc:348] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.485713 31933 raft_consensus.cc:374] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.485754 31933 raft_consensus.cc:729] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.486367 31932 tablet_bootstrap.cc:654] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.487666 31933 consensus_queue.cc:260] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.487821 31932 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:57.488149 31933 ts_tablet_manager.cc:1398] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:57.488847 31932 ts_tablet_manager.cc:1367] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:57.488932 31844 tablet_bootstrap.cc:492] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:57.489046 31844 ts_tablet_manager.cc:1367] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:57.489301 31932 raft_consensus.cc:348] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.489481 31932 raft_consensus.cc:374] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.489567 31932 raft_consensus.cc:729] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.489549 31844 raft_consensus.cc:348] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.489676 31844 raft_consensus.cc:374] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.489724 31844 raft_consensus.cc:729] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.489858 31844 consensus_queue.cc:260] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.489866 31932 consensus_queue.cc:260] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.490160 31844 ts_tablet_manager.cc:1398] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:57.490777 31932 ts_tablet_manager.cc:1398] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:57.490940 31932 tablet_bootstrap.cc:492] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:57.491688 31847 tablet_bootstrap.cc:492] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:57.491798 31847 ts_tablet_manager.cc:1367] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:56:57.491864 31932 tablet_bootstrap.cc:654] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.492908 31932 tablet_bootstrap.cc:492] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:57.492982 31932 ts_tablet_manager.cc:1367] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:57.493470 31932 raft_consensus.cc:348] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.493607 31932 raft_consensus.cc:374] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.493651 31932 raft_consensus.cc:729] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.493798 31932 consensus_queue.cc:260] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.494132 31932 ts_tablet_manager.cc:1398] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:57.494474 27276 consensus_queue.cc:237] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.495817 31847 raft_consensus.cc:348] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.495958 31847 raft_consensus.cc:374] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.496022 27425 raft_consensus.cc:1261] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:57.496022 31847 raft_consensus.cc:729] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.496199 31847 consensus_queue.cc:260] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.496500 31847 ts_tablet_manager.cc:1398] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:57.496549 24987 raft_consensus.cc:1261] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:57.496665 31847 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:57.497565 31847 tablet_bootstrap.cc:654] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.497718 31142 consensus_queue.cc:1035] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:57.498353 31746 consensus_queue.cc:1035] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:57.499887 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ef4152fef8749f0994a952095225134. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:57.500023 31847 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:57.500095 31847 ts_tablet_manager.cc:1367] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:57.500530 31847 raft_consensus.cc:348] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.500629 31847 raft_consensus.cc:374] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.500661 31847 raft_consensus.cc:729] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.500767 31847 consensus_queue.cc:260] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.501041 31847 ts_tablet_manager.cc:1398] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:57.501196 31847 tablet_bootstrap.cc:492] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:57.502323 31847 tablet_bootstrap.cc:654] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:57.503283 31746 raft_consensus.cc:2932] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:57.503522 31847 tablet_bootstrap.cc:492] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:57.503608 31847 ts_tablet_manager.cc:1367] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:57.504122 24987 raft_consensus.cc:2932] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:57.504822 31847 raft_consensus.cc:348] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.504914 31847 raft_consensus.cc:374] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:57.504948 31847 raft_consensus.cc:729] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.505056 31847 consensus_queue.cc:260] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.505329 31847 ts_tablet_manager.cc:1398] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:57.508924 24445 catalog_manager.cc:5484] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:57.509130 27424 raft_consensus.cc:2932] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:57.548465 30967 raft_consensus.cc:482] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:57.548578 30967 raft_consensus.cc:504] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.548907 30967 leader_election.cc:288] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:57.549397 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e40a2230d149cd91d389c9f2b451f6" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:57.549546 27276 raft_consensus.cc:2454] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:57.549597 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e40a2230d149cd91d389c9f2b451f6" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:57.549751 27424 raft_consensus.cc:2454] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:57.549881 26967 leader_election.cc:302] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:57.550067 30967 raft_consensus.cc:2781] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:57.550135 30967 raft_consensus.cc:482] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:57.550181 30967 raft_consensus.cc:3037] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.551458 30967 raft_consensus.cc:504] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.551972 30967 leader_election.cc:288] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:57.552417 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e40a2230d149cd91d389c9f2b451f6" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:57.552488 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e40a2230d149cd91d389c9f2b451f6" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:57.552560 27424 raft_consensus.cc:3037] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.552626 27276 raft_consensus.cc:3037] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.553647 27276 raft_consensus.cc:2454] T a1e40a2230d149cd91d389c9f2b451f6 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:57.553666 27424 raft_consensus.cc:2454] T a1e40a2230d149cd91d389c9f2b451f6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:57.553957 26967 leader_election.cc:302] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:57.554584 30967 raft_consensus.cc:2781] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:57.554673 30967 raft_consensus.cc:686] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:57.554818 30967 consensus_queue.cc:237] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.557432 24445 catalog_manager.cc:5484] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:57.567193 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:57.572058 27424 raft_consensus.cc:1261] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:57.572542 24594 raft_consensus.cc:1261] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:57.572973 31406 consensus_queue.cc:1035] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:57.573503 31406 consensus_queue.cc:1035] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:57.577109 24445 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 8d527149db554c0c8c2472d1374c0d1f (table rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc]). Expected version 1 got 0
I20240501 19:56:57.578477 31998 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:57.578506 24446 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 8d527149db554c0c8c2472d1374c0d1f (table rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc]). Expected version 1 got 0
I20240501 19:56:57.578809 31998 tablet.cc:1620] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:57.584223 24445 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 8d527149db554c0c8c2472d1374c0d1f (table rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc]). Expected version 1 got 0
I20240501 19:56:57.586407 31999 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:57.586768 31999 tablet.cc:1620] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:57.588762 31951 raft_consensus.cc:482] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:57.588876 31951 raft_consensus.cc:504] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.589293 31951 leader_election.cc:288] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:57.590790 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c67171da5e4b8bbfce0398461ae902" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:57.590898 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c67171da5e4b8bbfce0398461ae902" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:57.590950 27032 raft_consensus.cc:2454] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:57.591015 27424 raft_consensus.cc:2454] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:57.591300 27227 leader_election.cc:302] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:57.591739 24453 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 8d527149db554c0c8c2472d1374c0d1f (table rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc]). Expected version 1 got 0
I20240501 19:56:57.592113 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc] alter complete (version 1)
I20240501 19:56:57.592531 31951 raft_consensus.cc:2781] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:57.592589 31951 raft_consensus.cc:482] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:57.592646 31951 raft_consensus.cc:3037] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.593663 32000 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:57.593737 31951 raft_consensus.cc:504] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.594064 32000 tablet.cc:1620] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:57.594123 31951 leader_election.cc:288] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:57.594731 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c67171da5e4b8bbfce0398461ae902" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:57.594838 27032 raft_consensus.cc:3037] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.594861 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c67171da5e4b8bbfce0398461ae902" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:57.594972 27424 raft_consensus.cc:3037] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.595945 27032 raft_consensus.cc:2454] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:57.596036 27424 raft_consensus.cc:2454] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:57.596349 27227 leader_election.cc:302] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:57.596501 31951 raft_consensus.cc:2781] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:57.596616 31951 raft_consensus.cc:686] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:57.596963 31951 consensus_queue.cc:237] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:56:57.598591 31900 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:57.598683 31900 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.599094 31900 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:57.599627 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad3e4102cd14025b897ac7da10efe9e" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:57.599562 24447 catalog_manager.cc:5484] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:56:57.599727 27032 raft_consensus.cc:2454] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:57.600029 27096 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:56:57.600265 31900 raft_consensus.cc:2781] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:57.600318 31900 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:57.600353 31900 raft_consensus.cc:3037] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.600343 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad3e4102cd14025b897ac7da10efe9e" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:57.600457 27424 raft_consensus.cc:2454] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:56:57.601341 31900 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.601692 31900 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:57.603755 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad3e4102cd14025b897ac7da10efe9e" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:57.603858 27032 raft_consensus.cc:3037] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.604007 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad3e4102cd14025b897ac7da10efe9e" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:57.604120 27424 raft_consensus.cc:3037] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:57.604843 27032 raft_consensus.cc:2454] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:57.604974 27424 raft_consensus.cc:2454] T 3ad3e4102cd14025b897ac7da10efe9e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:56:57.605163 27096 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:56:57.605356 31900 raft_consensus.cc:2781] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:57.605445 31900 raft_consensus.cc:686] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:56:57.605567 31900 consensus_queue.cc:237] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.607838 24453 catalog_manager.cc:5484] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:57.645774 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.646212 27012 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.650849 27404 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:57.653762 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:57.658982 27267 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:57.662245 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.673422 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:57.697135 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:57.703779 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.704031 31998 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:57.704365 31998 tablet.cc:1620] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:56:57.704757 31999 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:57.705039 31999 tablet.cc:1620] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:57.705778 32000 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:57.706086 32000 tablet.cc:1620] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:57.711371 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=f291d8e4ffcb48c4a03652095856d1fc] alter complete (version 2)
I20240501 19:56:57.814254 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:57.820024 31998 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:57.820957 31999 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:57.821792 32000 tablet.cc:1722] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:57.825304 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=f291d8e4ffcb48c4a03652095856d1fc] alter complete (version 3)
I20240501 19:56:57.839610 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.840229 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:57.846971 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.850324 27012 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.850452 27404 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:57.854566 27267 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:57.869033 32077 ts_tablet_manager.cc:902] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:57.869519 32077 tablet_copy_client.cc:308] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.6:44579
I20240501 19:56:57.869932 27183 tablet_copy_service.cc:140] P 29d1a658e2b946e0b412e5fef82db9d9: Received BeginTabletCopySession request for tablet 4a4c18dfded64901bb4645af3dbe2dbc from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:50333)
I20240501 19:56:57.870038 27183 tablet_copy_service.cc:161] P 29d1a658e2b946e0b412e5fef82db9d9: Beginning new tablet copy session on tablet 4a4c18dfded64901bb4645af3dbe2dbc from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:50333: session id = 9d64cdfaeb2d46b29845160e395fc241-4a4c18dfded64901bb4645af3dbe2dbc
I20240501 19:56:57.870756 27183 tablet_copy_source_session.cc:205] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:57.875356 32077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4c18dfded64901bb4645af3dbe2dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.879113 32077 tablet_copy_client.cc:783] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:57.879417 32077 tablet_copy_client.cc:647] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:57.881876 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:57.885885 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.888847 32077 tablet_copy_client.cc:523] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:57.890362 32077 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:57.891263 32083 ts_tablet_manager.cc:902] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:57.891849 32083 tablet_copy_client.cc:308] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:56:57.892364 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet 0ef4152fef8749f0994a952095225134 from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:33371)
I20240501 19:56:57.892460 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet 0ef4152fef8749f0994a952095225134 from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:33371: session id = 29d1a658e2b946e0b412e5fef82db9d9-0ef4152fef8749f0994a952095225134
I20240501 19:56:57.893355 27314 tablet_copy_source_session.cc:205] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:57.894208 32083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef4152fef8749f0994a952095225134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.896482 32083 tablet_copy_client.cc:783] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:57.896703 32083 tablet_copy_client.cc:647] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:57.898869 32083 tablet_copy_client.cc:523] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:57.900971 32083 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:57.906026 32083 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:57.906240 32077 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:57.906447 32083 tablet_bootstrap.cc:492] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:56:57.906529 32077 tablet_bootstrap.cc:492] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:56:57.906698 32083 ts_tablet_manager.cc:1367] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240501 19:56:57.906698 32077 ts_tablet_manager.cc:1367] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 19:56:57.907387 32083 raft_consensus.cc:348] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.907311 32077 raft_consensus.cc:348] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:57.907608 32077 raft_consensus.cc:729] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.907732 32083 raft_consensus.cc:729] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.907773 32077 consensus_queue.cc:260] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:57.907899 32083 consensus_queue.cc:260] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:57.908152 32077 ts_tablet_manager.cc:1398] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:57.908979 27183 tablet_copy_service.cc:342] P 29d1a658e2b946e0b412e5fef82db9d9: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-4a4c18dfded64901bb4645af3dbe2dbc received from {username='slave'} at 127.17.65.3:50333
I20240501 19:56:57.909040 27183 tablet_copy_service.cc:433] P 29d1a658e2b946e0b412e5fef82db9d9: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-4a4c18dfded64901bb4645af3dbe2dbc on tablet 4a4c18dfded64901bb4645af3dbe2dbc with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:56:57.911296 32090 ts_tablet_manager.cc:902] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:56:57.911788 32090 tablet_copy_client.cc:308] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:56:57.912828 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet 0f1f7372e5ce42b284cf1179c72f578d from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:37455)
I20240501 19:56:57.912935 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet 0f1f7372e5ce42b284cf1179c72f578d from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:37455: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-0f1f7372e5ce42b284cf1179c72f578d
I20240501 19:56:57.913475 32083 ts_tablet_manager.cc:1398] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:56:57.914057 24746 tablet_copy_source_session.cc:205] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:57.914999 32090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1f7372e5ce42b284cf1179c72f578d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.916821 32090 tablet_copy_client.cc:783] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:57.916965 32090 tablet_copy_client.cc:647] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:57.917093 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-0ef4152fef8749f0994a952095225134 received from {username='slave'} at 127.17.65.6:33371
I20240501 19:56:57.917166 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-0ef4152fef8749f0994a952095225134 on tablet 0ef4152fef8749f0994a952095225134 with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:56:57.918210 32090 tablet_copy_client.cc:523] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:57.919741 32090 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:57.926221 24452 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:57.926365 24452 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:57.929741 24452 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 8d527149db554c0c8c2472d1374c0d1f
I20240501 19:56:57.930645 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d527149db554c0c8c2472d1374c0d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:57 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:57.930903 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d527149db554c0c8c2472d1374c0d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:57 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:56:57.931277 27404 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d527149db554c0c8c2472d1374c0d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:57 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:57.931986 32109 tablet_replica.cc:330] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:57.932139 32109 raft_consensus.cc:2229] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:57.932269 32109 raft_consensus.cc:2258] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.932518 32108 tablet_replica.cc:330] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:56:57.932667 32108 raft_consensus.cc:2229] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:57.932859 32108 raft_consensus.cc:2258] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.933470 32109 ts_tablet_manager.cc:1875] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:57.933529 32108 ts_tablet_manager.cc:1875] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:57.934711 32107 tablet_replica.cc:330] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:57.934911 32107 raft_consensus.cc:2229] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:57.935114 32107 raft_consensus.cc:2258] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.936082 32108 ts_tablet_manager.cc:1888] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:57.936190 32108 log.cc:1196] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/8d527149db554c0c8c2472d1374c0d1f
I20240501 19:56:57.936604 32108 ts_tablet_manager.cc:1909] T 8d527149db554c0c8c2472d1374c0d1f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:56:57.936618 32107 ts_tablet_manager.cc:1875] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:57.938104 32109 ts_tablet_manager.cc:1888] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:57.938227 32109 log.cc:1196] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/8d527149db554c0c8c2472d1374c0d1f
I20240501 19:56:57.938557 32109 ts_tablet_manager.cc:1909] T 8d527149db554c0c8c2472d1374c0d1f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:57.938731 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 8d527149db554c0c8c2472d1374c0d1f (table rebalancer_dynamic_table_new_name [id=f291d8e4ffcb48c4a03652095856d1fc]) successfully deleted
I20240501 19:56:57.940723 32107 ts_tablet_manager.cc:1888] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:57.940860 32107 log.cc:1196] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/8d527149db554c0c8c2472d1374c0d1f
I20240501 19:56:57.941210 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 8d527149db554c0c8c2472d1374c0d1f (table rebalancer_dynamic_table_new_name [id=f291d8e4ffcb48c4a03652095856d1fc]) successfully deleted
I20240501 19:56:57.941251 32107 ts_tablet_manager.cc:1909] T 8d527149db554c0c8c2472d1374c0d1f P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:57.941988 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 8d527149db554c0c8c2472d1374c0d1f (table rebalancer_dynamic_table_new_name [id=f291d8e4ffcb48c4a03652095856d1fc]) successfully deleted
I20240501 19:56:57.947223 32110 ts_tablet_manager.cc:902] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: Initiating tablet copy from peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:57.947839 32110 tablet_copy_client.cc:308] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.6:44579
I20240501 19:56:57.948861 27183 tablet_copy_service.cc:140] P 29d1a658e2b946e0b412e5fef82db9d9: Received BeginTabletCopySession request for tablet 2a44d1713369478191bfc30457bf61c1 from peer b9fd7c49d12b48a7ae73ab09876a0b82 ({username='slave'} at 127.17.65.4:38257)
I20240501 19:56:57.948977 27183 tablet_copy_service.cc:161] P 29d1a658e2b946e0b412e5fef82db9d9: Beginning new tablet copy session on tablet 2a44d1713369478191bfc30457bf61c1 from peer b9fd7c49d12b48a7ae73ab09876a0b82 at {username='slave'} at 127.17.65.4:38257: session id = b9fd7c49d12b48a7ae73ab09876a0b82-2a44d1713369478191bfc30457bf61c1
I20240501 19:56:57.949993 27183 tablet_copy_source_session.cc:205] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:57.951735 32090 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:57.952039 32090 tablet_bootstrap.cc:492] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:56:57.952242 32090 ts_tablet_manager.cc:1367] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.003s
I20240501 19:56:57.952692 32090 raft_consensus.cc:348] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:57.952806 32090 raft_consensus.cc:729] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.953279 32110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a44d1713369478191bfc30457bf61c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:57.954999 32090 consensus_queue.cc:260] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:57.955643 32090 ts_tablet_manager.cc:1398] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:57.956162 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-0f1f7372e5ce42b284cf1179c72f578d received from {username='slave'} at 127.17.65.5:37455
I20240501 19:56:57.956254 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-0f1f7372e5ce42b284cf1179c72f578d on tablet 0f1f7372e5ce42b284cf1179c72f578d with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:57.958026 32110 tablet_copy_client.cc:783] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:57.958386 32110 tablet_copy_client.cc:647] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:57.960326 32110 tablet_copy_client.cc:523] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:57.962127 32110 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:57.966507 32110 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:57.966771 32110 tablet_bootstrap.cc:492] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap complete.
I20240501 19:56:57.966946 32110 ts_tablet_manager.cc:1367] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:56:57.967342 32110 raft_consensus.cc:348] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:57.967456 32110 raft_consensus.cc:729] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:57.967600 32110 consensus_queue.cc:260] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:57.967890 32110 ts_tablet_manager.cc:1398] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:57.968410 27183 tablet_copy_service.cc:342] P 29d1a658e2b946e0b412e5fef82db9d9: Request end of tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-2a44d1713369478191bfc30457bf61c1 received from {username='slave'} at 127.17.65.4:38257
I20240501 19:56:57.968497 27183 tablet_copy_service.cc:433] P 29d1a658e2b946e0b412e5fef82db9d9: ending tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-2a44d1713369478191bfc30457bf61c1 on tablet 2a44d1713369478191bfc30457bf61c1 with peer b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:56:57.998253 32146 ts_tablet_manager.cc:902] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:56:57.998872 32146 tablet_copy_client.cc:308] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:56:57.999415 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 2e30faedce3742e0ba83de21059613d1 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:56477)
I20240501 19:56:57.999528 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 2e30faedce3742e0ba83de21059613d1 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:56477: session id = 4a6e54e86a9949e5970b42e8e32e7561-2e30faedce3742e0ba83de21059613d1
I20240501 19:56:58.000484 24616 tablet_copy_source_session.cc:205] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:58.001482 32146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e30faedce3742e0ba83de21059613d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.003940 32146 tablet_copy_client.cc:783] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:58.004146 32146 tablet_copy_client.cc:647] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:58.005831 32146 tablet_copy_client.cc:523] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:58.007179 32146 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:58.017997 31406 consensus_queue.cc:1035] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.023167 31899 consensus_queue.cc:1035] T 3ad3e4102cd14025b897ac7da10efe9e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.032609 24445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:58.033478 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:58.043617 32146 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:58.043973 32146 tablet_bootstrap.cc:492] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:56:58.044198 32146 ts_tablet_manager.cc:1367] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.000s
I20240501 19:56:58.044696 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6940ca429544a095031f04f31fd3bb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f]), partition=
I20240501 19:56:58.044766 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6940ca429544a095031f04f31fd3bb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f]), partition=
I20240501 19:56:58.044912 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6940ca429544a095031f04f31fd3bb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f]), partition=
I20240501 19:56:58.044970 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6940ca429544a095031f04f31fd3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.045024 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6940ca429544a095031f04f31fd3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.045153 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6940ca429544a095031f04f31fd3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.047842 32146 raft_consensus.cc:348] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.048065 32146 raft_consensus.cc:729] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.048290 32146 consensus_queue.cc:260] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.048699 32146 ts_tablet_manager.cc:1398] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:58.049225 32166 tablet_bootstrap.cc:492] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:58.049926 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-2e30faedce3742e0ba83de21059613d1 received from {username='slave'} at 127.17.65.8:56477
I20240501 19:56:58.050011 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-2e30faedce3742e0ba83de21059613d1 on tablet 2e30faedce3742e0ba83de21059613d1 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:56:58.052631 32166 tablet_bootstrap.cc:654] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.053234 30967 consensus_queue.cc:1035] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.055861 32166 tablet_bootstrap.cc:492] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:58.055963 32166 ts_tablet_manager.cc:1367] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:58.056231 24574 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:58.056449 32166 raft_consensus.cc:348] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.056551 32166 raft_consensus.cc:374] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.056587 32166 raft_consensus.cc:729] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.056730 32166 consensus_queue.cc:260] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.057025 32166 ts_tablet_manager.cc:1398] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.057308 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:58.058182 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:58.067158 32167 tablet_bootstrap.cc:492] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:58.068477 32167 tablet_bootstrap.cc:654] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.068707 24967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:58.070341 27267 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:58.071972 32170 tablet_bootstrap.cc:492] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:58.072320 27404 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:58.072969 32170 tablet_bootstrap.cc:654] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.074213 32167 tablet_bootstrap.cc:492] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:58.074362 32167 ts_tablet_manager.cc:1367] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:58.074983 32167 raft_consensus.cc:348] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.075191 32167 raft_consensus.cc:374] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.075269 32167 raft_consensus.cc:729] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.075436 32167 consensus_queue.cc:260] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.075747 32167 ts_tablet_manager.cc:1398] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.081499 30967 consensus_queue.cc:1035] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.082926 27012 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:58.111560 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:58.117843 27012 tablet_service.cc:1467] Processing CreateTablet for tablet f4f099923d754eb791bd4f567f4d2774 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.118206 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f099923d754eb791bd4f567f4d2774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.118688 24967 tablet_service.cc:1467] Processing CreateTablet for tablet f4f099923d754eb791bd4f567f4d2774 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.118780 24966 tablet_service.cc:1467] Processing CreateTablet for tablet d530d173786b4d3996e781e0c677abf4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.118932 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f099923d754eb791bd4f567f4d2774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.119102 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d530d173786b4d3996e781e0c677abf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.121026 27143 tablet_service.cc:1467] Processing CreateTablet for tablet f4f099923d754eb791bd4f567f4d2774 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.121121 32166 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:58.121290 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f099923d754eb791bd4f567f4d2774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.121821 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 5110e612369847268a739b03260eec35 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.122037 32166 tablet_bootstrap.cc:654] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.122058 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5110e612369847268a739b03260eec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.122112 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 5110e612369847268a739b03260eec35 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.122382 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5110e612369847268a739b03260eec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.125123 27404 tablet_service.cc:1467] Processing CreateTablet for tablet d530d173786b4d3996e781e0c677abf4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.125196 27012 tablet_service.cc:1467] Processing CreateTablet for tablet d530d173786b4d3996e781e0c677abf4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.125397 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d530d173786b4d3996e781e0c677abf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.125418 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d530d173786b4d3996e781e0c677abf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.126029 32170 tablet_bootstrap.cc:492] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:58.126109 32170 ts_tablet_manager.cc:1367] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.000s
I20240501 19:56:58.126688 32170 raft_consensus.cc:348] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.126799 32170 raft_consensus.cc:374] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.126832 32170 raft_consensus.cc:729] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.126968 32170 consensus_queue.cc:260] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.127931 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 5110e612369847268a739b03260eec35 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.127982 32177 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:58.128137 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5110e612369847268a739b03260eec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.128170 32170 ts_tablet_manager.cc:1398] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:58.128347 32170 tablet_bootstrap.cc:492] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:58.128865 32177 tablet_bootstrap.cc:654] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.129262 32170 tablet_bootstrap.cc:654] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.129973 32166 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:58.130059 32166 ts_tablet_manager.cc:1367] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:58.130137 32177 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:58.130213 32177 ts_tablet_manager.cc:1367] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:58.130542 32166 raft_consensus.cc:348] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.130650 32166 raft_consensus.cc:374] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.130614 32177 raft_consensus.cc:348] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.130683 32166 raft_consensus.cc:729] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.130728 32177 raft_consensus.cc:374] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.130759 32177 raft_consensus.cc:729] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.130818 32166 consensus_queue.cc:260] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.131047 32177 consensus_queue.cc:260] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.131106 32166 ts_tablet_manager.cc:1398] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.131291 32166 tablet_bootstrap.cc:492] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:58.132153 32166 tablet_bootstrap.cc:654] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.132143 32177 ts_tablet_manager.cc:1398] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:58.132306 32177 tablet_bootstrap.cc:492] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:58.132318 32178 tablet_bootstrap.cc:492] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:58.133680 32177 tablet_bootstrap.cc:654] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.133224 32178 tablet_bootstrap.cc:654] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.134791 32177 tablet_bootstrap.cc:492] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:58.134865 32177 ts_tablet_manager.cc:1367] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:58.135290 32177 raft_consensus.cc:348] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.135389 32177 raft_consensus.cc:374] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.135416 32177 raft_consensus.cc:729] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.135514 32177 consensus_queue.cc:260] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.135741 32177 ts_tablet_manager.cc:1398] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.138132 32166 tablet_bootstrap.cc:492] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:58.138221 32166 ts_tablet_manager.cc:1367] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:58.138612 32166 raft_consensus.cc:348] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.138706 32166 raft_consensus.cc:374] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.138746 32166 raft_consensus.cc:729] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.138870 32166 consensus_queue.cc:260] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.139171 32166 ts_tablet_manager.cc:1398] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.139954 32167 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:58.144093 32167 tablet_bootstrap.cc:654] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.145399 32167 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:58.145480 32167 ts_tablet_manager.cc:1367] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:58.145888 32167 raft_consensus.cc:348] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.145990 32167 raft_consensus.cc:374] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.146023 32167 raft_consensus.cc:729] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.146135 32167 consensus_queue.cc:260] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.152621 32167 ts_tablet_manager.cc:1398] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:58.153815 32167 tablet_bootstrap.cc:492] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:58.155511 32167 tablet_bootstrap.cc:654] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.156774 32167 tablet_bootstrap.cc:492] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:58.156898 32167 ts_tablet_manager.cc:1367] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:58.158046 32167 raft_consensus.cc:348] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.158221 32167 raft_consensus.cc:374] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.158293 32167 raft_consensus.cc:729] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.158479 32167 consensus_queue.cc:260] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.159417 32167 ts_tablet_manager.cc:1398] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:58.160008 32170 tablet_bootstrap.cc:492] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:58.160132 32170 ts_tablet_manager.cc:1367] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240501 19:56:58.160622 32170 raft_consensus.cc:348] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.160737 32170 raft_consensus.cc:374] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.160764 32170 raft_consensus.cc:729] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.160877 32170 consensus_queue.cc:260] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.161131 32170 ts_tablet_manager.cc:1398] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.163314 31951 consensus_queue.cc:1035] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.165776 32178 tablet_bootstrap.cc:492] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:58.165881 32178 ts_tablet_manager.cc:1367] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.001s
I20240501 19:56:58.166318 32178 raft_consensus.cc:348] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.166399 32178 raft_consensus.cc:374] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.166421 32178 raft_consensus.cc:729] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.166561 32178 consensus_queue.cc:260] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.166836 32178 ts_tablet_manager.cc:1398] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:58.167130 32178 tablet_bootstrap.cc:492] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:58.167981 32178 tablet_bootstrap.cc:654] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.169813 32178 tablet_bootstrap.cc:492] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:58.169958 32178 ts_tablet_manager.cc:1367] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:58.172106 31951 consensus_queue.cc:1035] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.175017 32178 raft_consensus.cc:348] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.176079 32178 raft_consensus.cc:374] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.176123 32178 raft_consensus.cc:729] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.176246 32178 consensus_queue.cc:260] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.176507 32178 ts_tablet_manager.cc:1398] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20240501 19:56:58.179141 31091 raft_consensus.cc:482] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:58.179231 31091 raft_consensus.cc:504] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.179594 31091 leader_election.cc:288] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:58.180759 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5110e612369847268a739b03260eec35" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:58.180837 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5110e612369847268a739b03260eec35" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:58.180897 27162 raft_consensus.cc:2454] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:58.180939 27425 raft_consensus.cc:2454] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:56:58.181177 24531 leader_election.cc:302] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:58.181416 31091 raft_consensus.cc:2781] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:58.181470 31091 raft_consensus.cc:482] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:58.181510 31091 raft_consensus.cc:3037] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.182549 31091 raft_consensus.cc:504] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.182869 31091 leader_election.cc:288] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:58.183408 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5110e612369847268a739b03260eec35" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:58.183434 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5110e612369847268a739b03260eec35" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:58.183516 27425 raft_consensus.cc:3037] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.183530 27162 raft_consensus.cc:3037] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.184419 27162 raft_consensus.cc:2454] T 5110e612369847268a739b03260eec35 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:58.184422 27425 raft_consensus.cc:2454] T 5110e612369847268a739b03260eec35 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:56:58.184830 24531 leader_election.cc:302] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:58.185326 31091 raft_consensus.cc:2781] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:58.185420 31091 raft_consensus.cc:686] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:58.185537 31091 consensus_queue.cc:237] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.188139 24445 catalog_manager.cc:5484] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:58.215768 32173 raft_consensus.cc:482] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:58.215902 32173 raft_consensus.cc:504] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.216315 32173 leader_election.cc:288] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:58.218068 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6940ca429544a095031f04f31fd3bb" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:56:58.218211 24594 raft_consensus.cc:2454] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:58.218364 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6940ca429544a095031f04f31fd3bb" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:58.218521 24986 raft_consensus.cc:2454] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:58.218866 26965 leader_election.cc:302] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:58.220966 32173 raft_consensus.cc:2781] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:58.221045 32173 raft_consensus.cc:482] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:58.221079 32173 raft_consensus.cc:3037] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.222131 32173 raft_consensus.cc:504] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.222476 32173 leader_election.cc:288] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:58.222972 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6940ca429544a095031f04f31fd3bb" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:56:58.223116 24594 raft_consensus.cc:3037] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.223073 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6940ca429544a095031f04f31fd3bb" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:58.223217 24986 raft_consensus.cc:3037] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.224110 24594 raft_consensus.cc:2454] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:58.224226 24986 raft_consensus.cc:2454] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:58.224494 26966 leader_election.cc:302] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:58.224833 32173 raft_consensus.cc:2781] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:58.224918 32173 raft_consensus.cc:686] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:58.225044 32173 consensus_queue.cc:237] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:58.228230 24446 catalog_manager.cc:5484] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:58.229044 31630 raft_consensus.cc:482] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:58.229144 31630 raft_consensus.cc:504] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.229524 31630 leader_election.cc:288] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:58.229944 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f099923d754eb791bd4f567f4d2774" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:58.230108 27162 raft_consensus.cc:2454] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:58.230659 24924 leader_election.cc:302] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:58.231285 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f099923d754eb791bd4f567f4d2774" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:58.231408 27032 raft_consensus.cc:2454] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:56:58.231998 31630 raft_consensus.cc:2781] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:58.232059 31630 raft_consensus.cc:482] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:58.232100 31630 raft_consensus.cc:3037] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.233176 31630 raft_consensus.cc:504] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.233513 31630 leader_election.cc:288] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:58.234031 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f099923d754eb791bd4f567f4d2774" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:58.234148 27162 raft_consensus.cc:3037] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.235122 27162 raft_consensus.cc:2454] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:58.235435 24924 leader_election.cc:302] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:58.235582 31630 raft_consensus.cc:2781] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:58.235780 31630 raft_consensus.cc:686] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:56:58.235888 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f099923d754eb791bd4f567f4d2774" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:58.235977 27032 raft_consensus.cc:3037] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.235937 31630 consensus_queue.cc:237] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:56:58.236915 27032 raft_consensus.cc:2454] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:56:58.239423 24447 catalog_manager.cc:5484] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:56:58.316516 24857 raft_consensus.cc:1203] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:58.320370 32173 raft_consensus.cc:482] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:58.320489 32173 raft_consensus.cc:504] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.320845 32173 leader_election.cc:288] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:58.321192 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d530d173786b4d3996e781e0c677abf4" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:58.321221 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d530d173786b4d3996e781e0c677abf4" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:58.321316 24986 raft_consensus.cc:2454] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:58.321373 27425 raft_consensus.cc:2454] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:58.321648 26967 leader_election.cc:302] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:58.321830 32173 raft_consensus.cc:2781] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:58.321887 32173 raft_consensus.cc:482] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:58.321913 32173 raft_consensus.cc:3037] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.322937 32173 raft_consensus.cc:504] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.323333 32173 leader_election.cc:288] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:58.323608 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d530d173786b4d3996e781e0c677abf4" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:58.323590 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d530d173786b4d3996e781e0c677abf4" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:58.323712 27425 raft_consensus.cc:3037] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.323711 24986 raft_consensus.cc:3037] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.324703 24986 raft_consensus.cc:2454] T d530d173786b4d3996e781e0c677abf4 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:58.324703 27425 raft_consensus.cc:2454] T d530d173786b4d3996e781e0c677abf4 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:58.324975 26967 leader_election.cc:302] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:58.325191 32173 raft_consensus.cc:2781] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:58.325263 32173 raft_consensus.cc:686] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:58.325394 32173 consensus_queue.cc:237] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.327487 24446 catalog_manager.cc:5484] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:58.339447 24986 raft_consensus.cc:1203] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:58.342061 24446 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:58.348982 24986 raft_consensus.cc:1261] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:58.349602 24594 raft_consensus.cc:1261] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:58.350142 30967 consensus_queue.cc:1035] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.352344 30967 consensus_queue.cc:1035] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.355065 24446 catalog_manager.cc:5576] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261) does not have the latest schema for tablet 2e6940ca429544a095031f04f31fd3bb (table rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f]). Expected version 1 got 0
I20240501 19:56:58.358081 24450 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 2e6940ca429544a095031f04f31fd3bb (table rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f]). Expected version 1 got 0
I20240501 19:56:58.362746 27012 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:58.363282 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:58.363623 24450 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 2e6940ca429544a095031f04f31fd3bb (table rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f]). Expected version 1 got 0
I20240501 19:56:58.364825 27405 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:58.365633 27267 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:58.378558 32250 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:58.379078 32250 tablet.cc:1620] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:58.388029 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:58.388764 27031 raft_consensus.cc:1203] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:58.389365 24574 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:58.390548 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f] alter complete (version 1)
I20240501 19:56:58.391225 24446 catalog_manager.cc:5576] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261) does not have the latest schema for tablet 2e6940ca429544a095031f04f31fd3bb (table rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f]). Expected version 1 got 0
I20240501 19:56:58.393368 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:58.400179 32251 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:58.400547 32251 tablet.cc:1620] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:58.412467 27142 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:58.436091 27162 raft_consensus.cc:1203] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:58.439023 32252 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:58.440670 32252 tablet.cc:1620] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:56:58.461979 27425 raft_consensus.cc:1203] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:58.497553 24446 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:58.503618 32250 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:58.503948 32250 tablet.cc:1620] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:58.506048 32251 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:58.506351 32251 tablet.cc:1620] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:56:58.508880 32252 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:58.509171 32252 tablet.cc:1620] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:56:58.513254 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=0b7e42e4830740349195a78f023ee94f] alter complete (version 2)
I20240501 19:56:58.557651 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:58.558785 27142 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:58.562256 27012 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:58.567504 24574 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:58.575192 27405 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:58.583196 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:58.584703 27267 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:58.617120 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:58.622438 32250 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:58.622911 32251 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:58.626230 32252 tablet.cc:1722] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:58.629585 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=0b7e42e4830740349195a78f023ee94f] alter complete (version 3)
I20240501 19:56:58.630038 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:58.665606 31882 consensus_queue.cc:1035] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.668867 32319 consensus_queue.cc:1035] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.718470 32232 consensus_queue.cc:1035] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.722764 31630 consensus_queue.cc:1035] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.732005 24446 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:58.732122 24446 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:58.734483 24446 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 2e6940ca429544a095031f04f31fd3bb
I20240501 19:56:58.735384 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6940ca429544a095031f04f31fd3bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:58 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:58.735730 32354 tablet_replica.cc:330] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:58.735953 32354 raft_consensus.cc:2229] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:58.736341 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6940ca429544a095031f04f31fd3bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:58 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:58.736552 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6940ca429544a095031f04f31fd3bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:58 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:56:58.736721 32354 raft_consensus.cc:2258] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:58.736907 32356 tablet_replica.cc:330] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:56:58.737092 32356 raft_consensus.cc:2229] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:58.737258 32356 raft_consensus.cc:2258] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:58.737396 32354 ts_tablet_manager.cc:1875] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:58.737911 32356 ts_tablet_manager.cc:1875] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:58.741082 32356 ts_tablet_manager.cc:1888] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:58.741362 32356 log.cc:1196] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/2e6940ca429544a095031f04f31fd3bb
I20240501 19:56:58.741442 32354 ts_tablet_manager.cc:1888] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:58.741518 32354 log.cc:1196] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/2e6940ca429544a095031f04f31fd3bb
I20240501 19:56:58.741677 32355 tablet_replica.cc:330] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:58.741782 32356 ts_tablet_manager.cc:1909] T 2e6940ca429544a095031f04f31fd3bb P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:56:58.741822 32355 raft_consensus.cc:2229] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:58.742050 32355 raft_consensus.cc:2258] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:58.742400 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 2e6940ca429544a095031f04f31fd3bb (table rebalancer_dynamic_table_new_name [id=0b7e42e4830740349195a78f023ee94f]) successfully deleted
I20240501 19:56:58.742486 32354 ts_tablet_manager.cc:1909] T 2e6940ca429544a095031f04f31fd3bb P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:56:58.743711 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 2e6940ca429544a095031f04f31fd3bb (table rebalancer_dynamic_table_new_name [id=0b7e42e4830740349195a78f023ee94f]) successfully deleted
I20240501 19:56:58.744001 32355 ts_tablet_manager.cc:1875] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:58.757936 27142 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:58.758791 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:58.762584 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:58.762780 27405 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:58.765074 27267 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:58.767449 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:58.775970 32173 consensus_queue.cc:1035] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.783862 30967 consensus_queue.cc:1035] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:58.800747 24574 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:58.804626 27012 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:58.806105 32355 ts_tablet_manager.cc:1888] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:58.808761 32355 log.cc:1196] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/2e6940ca429544a095031f04f31fd3bb
I20240501 19:56:58.809033 32355 ts_tablet_manager.cc:1909] T 2e6940ca429544a095031f04f31fd3bb P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:58.811246 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 2e6940ca429544a095031f04f31fd3bb (table rebalancer_dynamic_table_new_name [id=0b7e42e4830740349195a78f023ee94f]) successfully deleted
I20240501 19:56:58.836130 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:58.841943 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec]), partition=
I20240501 19:56:58.842060 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec]), partition=
I20240501 19:56:58.842218 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.842269 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.843284 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec]), partition=
I20240501 19:56:58.843518 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.845697 32381 tablet_bootstrap.cc:492] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:58.846093 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:58.849390 32381 tablet_bootstrap.cc:654] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.850759 32381 tablet_bootstrap.cc:492] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:58.850847 32381 ts_tablet_manager.cc:1367] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:58.851284 32381 raft_consensus.cc:348] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.851505 32381 raft_consensus.cc:374] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.851590 32381 raft_consensus.cc:729] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.851770 32381 consensus_queue.cc:260] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.852223 32381 ts_tablet_manager.cc:1398] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.854732 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 63b4288ae33d4e81a33648001a8c3f42 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.855011 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b4288ae33d4e81a33648001a8c3f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.855320 24705 tablet_service.cc:1467] Processing CreateTablet for tablet dca5963c18944efe8e5a87cd06d970b6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.855597 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca5963c18944efe8e5a87cd06d970b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.856470 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 37558df60b18476da2e153a4f9171266 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.856724 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37558df60b18476da2e153a4f9171266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.857686 32384 tablet_bootstrap.cc:492] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:56:58.857920 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 63b4288ae33d4e81a33648001a8c3f42 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.858134 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b4288ae33d4e81a33648001a8c3f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.858729 32384 tablet_bootstrap.cc:654] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.859000 27404 tablet_service.cc:1467] Processing CreateTablet for tablet dca5963c18944efe8e5a87cd06d970b6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.859241 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca5963c18944efe8e5a87cd06d970b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.861091 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 63b4288ae33d4e81a33648001a8c3f42 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.861440 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b4288ae33d4e81a33648001a8c3f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.861646 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 37558df60b18476da2e153a4f9171266 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.861907 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37558df60b18476da2e153a4f9171266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.861930 27010 tablet_service.cc:1467] Processing CreateTablet for tablet dca5963c18944efe8e5a87cd06d970b6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.862164 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca5963c18944efe8e5a87cd06d970b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.862692 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 37558df60b18476da2e153a4f9171266 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c2451a10f568427881412a162ce73741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:58.862901 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37558df60b18476da2e153a4f9171266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:58.864001 32384 tablet_bootstrap.cc:492] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:56:58.864085 32384 ts_tablet_manager.cc:1367] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 19:56:58.864627 32381 tablet_bootstrap.cc:492] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:58.864720 32384 raft_consensus.cc:348] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.864929 32384 raft_consensus.cc:374] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.865017 32384 raft_consensus.cc:729] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.865331 32384 consensus_queue.cc:260] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.865507 32385 tablet_bootstrap.cc:492] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:58.865821 32384 ts_tablet_manager.cc:1398] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:58.865949 32382 tablet_bootstrap.cc:492] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:58.866500 32381 tablet_bootstrap.cc:654] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.867856 32385 tablet_bootstrap.cc:654] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.868008 32387 tablet_bootstrap.cc:492] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:58.869140 32382 tablet_bootstrap.cc:654] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.869458 32387 tablet_bootstrap.cc:654] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.869815 32385 tablet_bootstrap.cc:492] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:58.869856 32380 tablet_bootstrap.cc:492] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:58.869951 32385 ts_tablet_manager.cc:1367] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:58.870586 32387 tablet_bootstrap.cc:492] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:58.870599 32385 raft_consensus.cc:348] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.870769 32380 tablet_bootstrap.cc:654] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.870782 32385 raft_consensus.cc:374] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.871243 32385 raft_consensus.cc:729] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.871256 32387 ts_tablet_manager.cc:1367] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:58.871780 32387 raft_consensus.cc:348] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.871899 32387 raft_consensus.cc:374] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.871912 32382 tablet_bootstrap.cc:492] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:58.871963 32387 raft_consensus.cc:729] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.871992 32382 ts_tablet_manager.cc:1367] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:58.872088 32387 consensus_queue.cc:260] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.871726 32385 consensus_queue.cc:260] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.872571 32387 ts_tablet_manager.cc:1398] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.872547 32382 raft_consensus.cc:348] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.872715 32382 raft_consensus.cc:374] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.872793 32382 raft_consensus.cc:729] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.872813 32385 ts_tablet_manager.cc:1398] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:58.872982 32382 consensus_queue.cc:260] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.873100 32385 tablet_bootstrap.cc:492] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:56:58.874939 32381 tablet_bootstrap.cc:492] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:58.875056 32381 ts_tablet_manager.cc:1367] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 19:56:58.875519 32381 raft_consensus.cc:348] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.875630 32381 raft_consensus.cc:374] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.875665 32381 raft_consensus.cc:729] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.875725 32382 ts_tablet_manager.cc:1398] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:58.875777 32381 consensus_queue.cc:260] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.876075 32381 ts_tablet_manager.cc:1398] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:58.876250 32381 tablet_bootstrap.cc:492] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:58.876793 32382 tablet_bootstrap.cc:492] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:58.877171 32381 tablet_bootstrap.cc:654] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.878479 32381 tablet_bootstrap.cc:492] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:58.878544 32381 ts_tablet_manager.cc:1367] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:58.878937 32381 raft_consensus.cc:348] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.879024 32381 raft_consensus.cc:374] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.879392 32381 raft_consensus.cc:729] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.879591 32382 tablet_bootstrap.cc:654] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.879556 32381 consensus_queue.cc:260] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.879926 32381 ts_tablet_manager.cc:1398] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.880728 32380 tablet_bootstrap.cc:492] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:58.881104 32380 ts_tablet_manager.cc:1367] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:58.881889 32380 raft_consensus.cc:348] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.882411 32385 tablet_bootstrap.cc:654] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.882467 32382 tablet_bootstrap.cc:492] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:58.882544 32380 raft_consensus.cc:374] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.882642 32380 raft_consensus.cc:729] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.882838 32380 consensus_queue.cc:260] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.883561 32385 tablet_bootstrap.cc:492] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:56:58.883651 32385 ts_tablet_manager.cc:1367] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:58.884086 32385 raft_consensus.cc:348] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.884152 32380 ts_tablet_manager.cc:1398] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:58.884198 32385 raft_consensus.cc:374] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.884230 32385 raft_consensus.cc:729] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.884323 32380 tablet_bootstrap.cc:492] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:58.884339 32385 consensus_queue.cc:260] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.884608 32385 ts_tablet_manager.cc:1398] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:58.884967 32382 ts_tablet_manager.cc:1367] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:58.886438 32380 tablet_bootstrap.cc:654] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.888589 32382 raft_consensus.cc:348] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.888687 32382 raft_consensus.cc:374] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.888721 32382 raft_consensus.cc:729] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.888844 24594 consensus_queue.cc:237] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.888888 32382 consensus_queue.cc:260] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.889781 32382 ts_tablet_manager.cc:1398] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:58.889966 32382 tablet_bootstrap.cc:492] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:58.890205 27425 raft_consensus.cc:1261] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:58.890226 27163 raft_consensus.cc:1261] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:58.890669 31882 consensus_queue.cc:1035] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:58.890986 31882 consensus_queue.cc:1035] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:58.891765 32380 tablet_bootstrap.cc:492] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:58.891896 32380 ts_tablet_manager.cc:1367] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:58.892379 32380 raft_consensus.cc:348] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.892724 32382 tablet_bootstrap.cc:654] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:58.893276 32380 raft_consensus.cc:374] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.893383 32380 raft_consensus.cc:729] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.893361 31091 raft_consensus.cc:2932] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:58.893584 32380 consensus_queue.cc:260] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.895135 32380 ts_tablet_manager.cc:1398] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:58.895860 32382 tablet_bootstrap.cc:492] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:58.895990 32382 ts_tablet_manager.cc:1367] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:58.896229 27163 raft_consensus.cc:2932] T caf81a72d0304b8aa611d4c1eab5ac26 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:58.896502 32382 raft_consensus.cc:348] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.896618 32382 raft_consensus.cc:374] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:58.896701 32382 raft_consensus.cc:729] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:58.896859 32382 consensus_queue.cc:260] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:58.897799 32382 ts_tablet_manager.cc:1398] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:58.898661 24453 catalog_manager.cc:5484] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:58.903419 27425 raft_consensus.cc:2932] T caf81a72d0304b8aa611d4c1eab5ac26 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:58.905062 27010 tablet_service.cc:1514] Processing DeleteTablet for tablet caf81a72d0304b8aa611d4c1eab5ac26 with delete_type TABLET_DATA_TOMBSTONED (TS 3b5968cebd0b4294898fdd89b7a5b3a1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:58.905371 32355 tablet_replica.cc:330] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:58.905493 32355 raft_consensus.cc:2229] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:58.905575 32355 raft_consensus.cc:2258] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:58.905994 32355 ts_tablet_manager.cc:1875] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:58.908488 32355 ts_tablet_manager.cc:1888] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:56:58.908671 32355 log.cc:1196] T caf81a72d0304b8aa611d4c1eab5ac26 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/caf81a72d0304b8aa611d4c1eab5ac26
I20240501 19:56:58.909377 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet caf81a72d0304b8aa611d4c1eab5ac26 (table rebalancer_extra_table_0 [id=e48b2299dd574fcab52da0f4ecad97b9]) successfully deleted
I20240501 19:56:58.964655 32173 raft_consensus.cc:482] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:58.964795 32173 raft_consensus.cc:504] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.965126 32173 leader_election.cc:288] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:58.965437 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f75f9cdb9ff4191b82f9cd977a7ea3f" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:56:58.965468 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f75f9cdb9ff4191b82f9cd977a7ea3f" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:58.965581 27425 raft_consensus.cc:2454] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:58.965632 24857 raft_consensus.cc:2454] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:58.965860 26967 leader_election.cc:302] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:58.966046 32173 raft_consensus.cc:2781] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:58.966095 32173 raft_consensus.cc:482] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:58.966130 32173 raft_consensus.cc:3037] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.967430 32173 raft_consensus.cc:504] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.967792 32173 leader_election.cc:288] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:56:58.969362 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f75f9cdb9ff4191b82f9cd977a7ea3f" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:56:58.969499 27425 raft_consensus.cc:3037] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.970542 27425 raft_consensus.cc:2454] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:58.972309 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f75f9cdb9ff4191b82f9cd977a7ea3f" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:58.972430 24857 raft_consensus.cc:3037] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.973474 24857 raft_consensus.cc:2454] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:58.973847 26966 leader_election.cc:302] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:56:58.974304 32173 raft_consensus.cc:2781] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:58.974387 32173 raft_consensus.cc:686] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:58.974530 32173 consensus_queue.cc:237] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.978855 24452 catalog_manager.cc:5484] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:58.991354 30935 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:58.991489 30935 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.991940 30935 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:58.992561 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37558df60b18476da2e153a4f9171266" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:58.992738 27163 raft_consensus.cc:2454] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:58.992986 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37558df60b18476da2e153a4f9171266" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:56:58.993036 27361 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:58.993134 27276 raft_consensus.cc:2454] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:58.993192 30935 raft_consensus.cc:2781] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:58.993258 30935 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:58.993297 30935 raft_consensus.cc:3037] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.994484 30935 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.994850 30935 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:56:58.995226 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37558df60b18476da2e153a4f9171266" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:58.995334 27163 raft_consensus.cc:3037] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.995328 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37558df60b18476da2e153a4f9171266" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:56:58.995453 27276 raft_consensus.cc:3037] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:58.996335 27163 raft_consensus.cc:2454] T 37558df60b18476da2e153a4f9171266 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:58.996340 27276 raft_consensus.cc:2454] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:58.996629 27361 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:56:58.996798 30935 raft_consensus.cc:2781] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:58.996881 30935 raft_consensus.cc:686] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:58.997028 30935 consensus_queue.cc:237] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:58.999281 24451 catalog_manager.cc:5484] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:56:59.043105 32173 raft_consensus.cc:482] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:59.043215 32173 raft_consensus.cc:504] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.043569 32173 leader_election.cc:288] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:59.043903 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b4288ae33d4e81a33648001a8c3f42" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:56:59.044055 27163 raft_consensus.cc:2454] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:59.044023 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b4288ae33d4e81a33648001a8c3f42" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:56:59.044186 24857 raft_consensus.cc:2454] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:56:59.044322 26967 leader_election.cc:302] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:56:59.044525 32173 raft_consensus.cc:2781] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:59.044588 32173 raft_consensus.cc:482] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:59.044633 32173 raft_consensus.cc:3037] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.045445 32173 raft_consensus.cc:504] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.045758 32173 leader_election.cc:288] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:56:59.046033 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b4288ae33d4e81a33648001a8c3f42" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:56:59.046034 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b4288ae33d4e81a33648001a8c3f42" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:59.046128 27163 raft_consensus.cc:3037] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.046135 24857 raft_consensus.cc:3037] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.046880 27163 raft_consensus.cc:2454] T 63b4288ae33d4e81a33648001a8c3f42 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:59.046949 24857 raft_consensus.cc:2454] T 63b4288ae33d4e81a33648001a8c3f42 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:56:59.047153 26967 leader_election.cc:302] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:56:59.047295 32173 raft_consensus.cc:2781] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:59.047381 32173 raft_consensus.cc:686] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:56:59.047524 32173 consensus_queue.cc:237] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.049432 24451 catalog_manager.cc:5484] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:59.098491 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:59.102669 27425 raft_consensus.cc:1261] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:59.102784 24857 raft_consensus.cc:1261] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:59.103176 32173 consensus_queue.cc:1035] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.103490 30967 consensus_queue.cc:1035] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.106133 24451 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (table rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec]). Expected version 1 got 0
I20240501 19:56:59.107187 24446 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (table rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec]). Expected version 1 got 0
I20240501 19:56:59.108294 32250 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:59.108577 32250 tablet.cc:1620] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:59.110878 32455 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:59.111188 32455 tablet.cc:1620] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:56:59.111303 24451 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (table rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec]). Expected version 1 got 0
I20240501 19:56:59.112251 24447 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (table rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec]). Expected version 1 got 0
I20240501 19:56:59.113353 32456 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:59.113691 32456 tablet.cc:1620] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:59.115746 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec] alter complete (version 1)
I20240501 19:56:59.217392 24447 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:59.222746 32250 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:59.223107 32250 tablet.cc:1620] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:59.223553 32455 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:59.223863 32455 tablet.cc:1620] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:56:59.224151 32456 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:59.225271 32456 tablet.cc:1620] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:56:59.228327 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=04690d7da3d9487f842e4c62b43726ec] alter complete (version 2)
I20240501 19:56:59.333281 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:59.339581 32250 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:59.341322 31616 raft_consensus.cc:482] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:59.341436 31616 raft_consensus.cc:504] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.341807 31616 leader_election.cc:288] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:59.342048 32455 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:59.342175 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca5963c18944efe8e5a87cd06d970b6" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:56:59.342336 24725 raft_consensus.cc:2454] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:59.342432 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca5963c18944efe8e5a87cd06d970b6" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:59.342551 27032 raft_consensus.cc:2454] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:56:59.342629 27359 leader_election.cc:302] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:59.342906 31616 raft_consensus.cc:2781] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:59.342967 31616 raft_consensus.cc:482] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:59.343006 31616 raft_consensus.cc:3037] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.343977 32456 tablet.cc:1722] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:59.344053 31616 raft_consensus.cc:504] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.344398 31616 leader_election.cc:288] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:59.344996 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca5963c18944efe8e5a87cd06d970b6" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:59.345110 24725 raft_consensus.cc:3037] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.346120 24725 raft_consensus.cc:2454] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:59.347141 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=04690d7da3d9487f842e4c62b43726ec] alter complete (version 3)
I20240501 19:56:59.347183 27359 leader_election.cc:302] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:56:59.347640 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca5963c18944efe8e5a87cd06d970b6" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:59.347761 27032 raft_consensus.cc:3037] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.348675 27032 raft_consensus.cc:2454] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:56:59.349326 31616 raft_consensus.cc:2781] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:59.349438 31616 raft_consensus.cc:686] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:56:59.349578 31616 consensus_queue.cc:237] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.351569 24446 catalog_manager.cc:5484] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:59.393326 27010 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:59.394766 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:59.404178 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:59.409087 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:59.413836 27267 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:59.420079 31616 consensus_queue.cc:1035] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.432601 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:59.439139 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:59.452971 24446 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:59.453119 24446 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:59.455829 24446 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f
I20240501 19:56:59.456837 27010 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:59 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:56:59.457031 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:59 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:56:59.457091 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:59 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:56:59.457624 31616 consensus_queue.cc:1035] T 37558df60b18476da2e153a4f9171266 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.459146 32482 tablet_replica.cc:330] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:56:59.459380 32482 raft_consensus.cc:2229] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:59.459599 32482 raft_consensus.cc:2258] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:59.461704 30967 consensus_queue.cc:1035] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.465376 32482 ts_tablet_manager.cc:1875] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:59.465806 24574 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:59.467166 32480 tablet_replica.cc:330] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:56:59.467316 32480 raft_consensus.cc:2229] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:59.467538 32480 raft_consensus.cc:2258] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:59.467554 32482 ts_tablet_manager.cc:1888] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:59.467622 32482 log.cc:1196] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/9f75f9cdb9ff4191b82f9cd977a7ea3f
I20240501 19:56:59.467885 32482 ts_tablet_manager.cc:1909] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:56:59.471954 32481 tablet_replica.cc:330] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:56:59.472172 32481 raft_consensus.cc:2229] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:59.472304 32481 raft_consensus.cc:2258] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:59.472841 32481 ts_tablet_manager.cc:1875] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:59.474880 32481 ts_tablet_manager.cc:1888] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:59.474956 32481 log.cc:1196] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/9f75f9cdb9ff4191b82f9cd977a7ea3f
I20240501 19:56:59.475948 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (table rebalancer_dynamic_table_new_name [id=04690d7da3d9487f842e4c62b43726ec]) successfully deleted
I20240501 19:56:59.476219 32481 ts_tablet_manager.cc:1909] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:56:59.477298 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (table rebalancer_dynamic_table_new_name [id=04690d7da3d9487f842e4c62b43726ec]) successfully deleted
I20240501 19:56:59.478524 30967 consensus_queue.cc:1035] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.481333 32480 ts_tablet_manager.cc:1875] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:59.484103 32480 ts_tablet_manager.cc:1888] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:59.484205 32480 log.cc:1196] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/9f75f9cdb9ff4191b82f9cd977a7ea3f
I20240501 19:56:59.484472 32480 ts_tablet_manager.cc:1909] T 9f75f9cdb9ff4191b82f9cd977a7ea3f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:56:59.485090 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 9f75f9cdb9ff4191b82f9cd977a7ea3f (table rebalancer_dynamic_table_new_name [id=04690d7da3d9487f842e4c62b43726ec]) successfully deleted
I20240501 19:56:59.510608 27163 consensus_queue.cc:237] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:56:59.511776 24856 raft_consensus.cc:1261] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:59.511859 27425 raft_consensus.cc:1261] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:59.512148 32504 consensus_queue.cc:1035] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:59.512341 32406 consensus_queue.cc:1035] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:59.514147 32507 raft_consensus.cc:2932] T afc479e18b5e4efa8a8fa113db3dbf1b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:59.514453 27425 raft_consensus.cc:2932] T afc479e18b5e4efa8a8fa113db3dbf1b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:59.514541 24856 raft_consensus.cc:2932] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:59.517463 24451 catalog_manager.cc:5484] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:56:59.520076 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet afc479e18b5e4efa8a8fa113db3dbf1b with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48994
I20240501 19:56:59.520485 32508 tablet_replica.cc:330] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:56:59.520677 32508 raft_consensus.cc:2229] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:59.520968 32508 raft_consensus.cc:2258] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:59.521437 32508 ts_tablet_manager.cc:1875] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:59.523852 32508 ts_tablet_manager.cc:1888] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:56:59.523933 32508 log.cc:1196] T afc479e18b5e4efa8a8fa113db3dbf1b P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/afc479e18b5e4efa8a8fa113db3dbf1b
I20240501 19:56:59.524536 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet afc479e18b5e4efa8a8fa113db3dbf1b (table rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]) successfully deleted
I20240501 19:56:59.559724 24447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:59.566399 27267 tablet_service.cc:1467] Processing CreateTablet for tablet ed08a34bb39741dcbeab8d2073e6fce1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba6e5b25739548a1a13cd200757cecdb]), partition=
I20240501 19:56:59.566601 27010 tablet_service.cc:1467] Processing CreateTablet for tablet ed08a34bb39741dcbeab8d2073e6fce1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba6e5b25739548a1a13cd200757cecdb]), partition=
I20240501 19:56:59.566743 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed08a34bb39741dcbeab8d2073e6fce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.566874 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed08a34bb39741dcbeab8d2073e6fce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.569252 24966 tablet_service.cc:1467] Processing CreateTablet for tablet ed08a34bb39741dcbeab8d2073e6fce1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba6e5b25739548a1a13cd200757cecdb]), partition=
I20240501 19:56:59.569494 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed08a34bb39741dcbeab8d2073e6fce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.570619 32551 tablet_bootstrap.cc:492] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:59.571086 32552 tablet_bootstrap.cc:492] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:56:59.572060 32552 tablet_bootstrap.cc:654] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.572613 32550 tablet_bootstrap.cc:492] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:59.573601 32550 tablet_bootstrap.cc:654] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.574339 32551 tablet_bootstrap.cc:654] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.576062 32550 tablet_bootstrap.cc:492] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:59.576154 32550 ts_tablet_manager.cc:1367] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:59.576325 32551 tablet_bootstrap.cc:492] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:59.576395 32551 ts_tablet_manager.cc:1367] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:59.576617 32550 raft_consensus.cc:348] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.576721 32550 raft_consensus.cc:374] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.576750 32550 raft_consensus.cc:729] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.576817 32551 raft_consensus.cc:348] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.576851 32550 consensus_queue.cc:260] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.576911 32551 raft_consensus.cc:374] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.576939 32551 raft_consensus.cc:729] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.577034 32551 consensus_queue.cc:260] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.577123 32550 ts_tablet_manager.cc:1398] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.577275 32551 ts_tablet_manager.cc:1398] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.594468 32552 tablet_bootstrap.cc:492] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:56:59.594578 32552 ts_tablet_manager.cc:1367] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 19:56:59.594995 32552 raft_consensus.cc:348] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.595113 32552 raft_consensus.cc:374] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.595140 32552 raft_consensus.cc:729] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.595242 32552 consensus_queue.cc:260] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.595510 32552 ts_tablet_manager.cc:1398] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.636396 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:59.638496 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:59.639195 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:59.639968 27010 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:59.647697 27267 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:59.652137 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:59.654628 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:59.682715 24574 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:59.702549 31952 raft_consensus.cc:482] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:59.702675 31952 raft_consensus.cc:504] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.703087 31952 leader_election.cc:288] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:59.703946 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed08a34bb39741dcbeab8d2073e6fce1" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:56:59.704095 24986 raft_consensus.cc:2454] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:59.704072 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed08a34bb39741dcbeab8d2073e6fce1" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:56:59.704210 27032 raft_consensus.cc:2454] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:56:59.704442 27228 leader_election.cc:302] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:56:59.704768 31952 raft_consensus.cc:2781] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:59.704838 31952 raft_consensus.cc:482] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:59.704885 31952 raft_consensus.cc:3037] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.705926 31952 raft_consensus.cc:504] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.706246 31952 leader_election.cc:288] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:56:59.706586 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed08a34bb39741dcbeab8d2073e6fce1" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:59.706707 27032 raft_consensus.cc:3037] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.707242 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed08a34bb39741dcbeab8d2073e6fce1" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:56:59.707346 24986 raft_consensus.cc:3037] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.707784 27032 raft_consensus.cc:2454] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:59.708304 24986 raft_consensus.cc:2454] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:56:59.708865 27227 leader_election.cc:302] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:56:59.709038 31952 raft_consensus.cc:2781] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:59.709120 31952 raft_consensus.cc:686] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:56:59.709247 31952 consensus_queue.cc:237] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.711308 24446 catalog_manager.cc:5484] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:59.725126 27163 consensus_queue.cc:237] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:56:59.726258 24987 raft_consensus.cc:1261] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:59.726263 27032 raft_consensus.cc:1261] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:59.726642 32504 consensus_queue.cc:1035] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:59.726801 32507 consensus_queue.cc:1035] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:59.728600 32504 raft_consensus.cc:2932] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:59.729359 24987 raft_consensus.cc:2932] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:59.730554 27032 raft_consensus.cc:2932] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:59.731521 24446 catalog_manager.cc:5484] T 2a44d1713369478191bfc30457bf61c1 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:56:59.736619 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a44d1713369478191bfc30457bf61c1 with delete_type TABLET_DATA_TOMBSTONED (TS 53e34bb09d8e41129a5683079ff63610 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49040
I20240501 19:56:59.737542 32591 tablet_replica.cc:330] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:56:59.737692 32591 raft_consensus.cc:2229] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:59.737818 32591 raft_consensus.cc:2258] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:59.738240 32591 ts_tablet_manager.cc:1875] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:59.740720 32591 ts_tablet_manager.cc:1888] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:56:59.740794 32591 log.cc:1196] T 2a44d1713369478191bfc30457bf61c1 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/2a44d1713369478191bfc30457bf61c1
I20240501 19:56:59.741585 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 2a44d1713369478191bfc30457bf61c1 (table rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]) successfully deleted
I20240501 19:56:59.809178 31616 consensus_queue.cc:1035] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.813138 30935 consensus_queue.cc:1035] T dca5963c18944efe8e5a87cd06d970b6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.839541 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:59.845806 27032 raft_consensus.cc:1261] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:59.846383 24987 raft_consensus.cc:1261] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:59.847273 31951 consensus_queue.cc:1035] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.848255 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:59.848680 31951 consensus_queue.cc:1035] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:59.853705 24448 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet ed08a34bb39741dcbeab8d2073e6fce1 (table rebalancer_dynamic_table [id=ba6e5b25739548a1a13cd200757cecdb]). Expected version 1 got 0
I20240501 19:56:59.854928 24449 catalog_manager.cc:5576] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261) does not have the latest schema for tablet ed08a34bb39741dcbeab8d2073e6fce1 (table rebalancer_dynamic_table [id=ba6e5b25739548a1a13cd200757cecdb]). Expected version 1 got 0
I20240501 19:56:59.854959 32648 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:59.855369 32648 tablet.cc:1620] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:59.858839 24837 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:59.858963 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:59.860431 27267 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:59.862175 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:59.862221 27010 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:59.864024 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=ba6e5b25739548a1a13cd200757cecdb] alter complete (version 1)
I20240501 19:56:59.868263 24450 catalog_manager.cc:5576] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261) does not have the latest schema for tablet ed08a34bb39741dcbeab8d2073e6fce1 (table rebalancer_dynamic_table [id=ba6e5b25739548a1a13cd200757cecdb]). Expected version 1 got 0
I20240501 19:56:59.888617 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:59.889029 32647 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:59.890518 32647 tablet.cc:1620] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:59.899237 24574 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:59.907642 32649 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:59.908169 32649 tablet.cc:1620] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:56:59.931941 24725 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14"
dest_uuid: "9081292597be4a72a42b216fddcd7971"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41724
I20240501 19:56:59.932085 24725 raft_consensus.cc:595] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:59.938122 32253 raft_consensus.cc:979] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: : Instructing follower bbf7735f82ca4f65a5170ac7909faa47 to start an election
I20240501 19:56:59.938220 32253 raft_consensus.cc:1067] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Signalling peer bbf7735f82ca4f65a5170ac7909faa47 to start an election
I20240501 19:56:59.938846 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:59.940627 24594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14"
dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
from {username='slave'} at 127.17.65.2:47433
I20240501 19:56:59.940773 24594 raft_consensus.cc:482] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:59.940827 24594 raft_consensus.cc:3037] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:59.941830 24594 raft_consensus.cc:504] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.942250 24594 leader_election.cc:288] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:56:59.942618 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:56:59.942726 24857 raft_consensus.cc:3037] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:59.943289 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:56:59.943400 27032 raft_consensus.cc:3037] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:59.944492 27032 raft_consensus.cc:2454] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 2.
I20240501 19:56:59.945386 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 2e37eadc7d034f11b68349f255227298 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.945518 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 2e37eadc7d034f11b68349f255227298 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.945714 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e37eadc7d034f11b68349f255227298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.945760 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e37eadc7d034f11b68349f255227298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.945739 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 54e20f102ed94db78eb4e735b843df24 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.946027 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e20f102ed94db78eb4e735b843df24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.946909 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 54e20f102ed94db78eb4e735b843df24 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.947198 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e20f102ed94db78eb4e735b843df24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.948483 27010 tablet_service.cc:1467] Processing CreateTablet for tablet 2e37eadc7d034f11b68349f255227298 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.948709 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e37eadc7d034f11b68349f255227298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.948918 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224ae792734e4d1ab3c28e20d82fdf14" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:56:59.949035 24725 raft_consensus.cc:3032] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:59.949074 24725 raft_consensus.cc:729] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:56:59.949223 24725 consensus_queue.cc:260] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.949389 24725 raft_consensus.cc:3037] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:59.949555 24574 tablet_service.cc:1467] Processing CreateTablet for tablet abfa17e323af4bbcb123de23ad1102e7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.949831 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfa17e323af4bbcb123de23ad1102e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.950443 24725 raft_consensus.cc:2454] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 2.
I20240501 19:56:59.950788 24529 leader_election.cc:302] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:56:59.950817 27011 tablet_service.cc:1467] Processing CreateTablet for tablet abfa17e323af4bbcb123de23ad1102e7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.950978 32476 raft_consensus.cc:2781] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:59.951022 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfa17e323af4bbcb123de23ad1102e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.951051 32476 raft_consensus.cc:686] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:56:59.951215 27267 tablet_service.cc:1467] Processing CreateTablet for tablet abfa17e323af4bbcb123de23ad1102e7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.951287 32476 consensus_queue.cc:237] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.951437 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfa17e323af4bbcb123de23ad1102e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.952296 32551 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:59.952963 32550 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:59.953311 32551 tablet_bootstrap.cc:654] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.953913 32550 tablet_bootstrap.cc:654] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.954515 32551 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:59.954592 32551 ts_tablet_manager.cc:1367] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:59.955132 32551 raft_consensus.cc:348] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:59.955209 32551 raft_consensus.cc:374] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.955229 32551 raft_consensus.cc:729] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.955325 32551 consensus_queue.cc:260] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:59.955534 24857 raft_consensus.cc:2454] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 2.
I20240501 19:56:59.955619 32551 ts_tablet_manager.cc:1398] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.955767 32551 tablet_bootstrap.cc:492] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:59.956506 32551 tablet_bootstrap.cc:654] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.957854 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 54e20f102ed94db78eb4e735b843df24 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.958127 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e20f102ed94db78eb4e735b843df24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.959754 24450 catalog_manager.cc:5484] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 1 to 2, leader changed from 9081292597be4a72a42b216fddcd7971 (127.17.65.2) to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 2 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:56:59.962357 32551 tablet_bootstrap.cc:492] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:59.962457 32551 ts_tablet_manager.cc:1367] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:59.962905 32551 raft_consensus.cc:348] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.963014 32551 raft_consensus.cc:374] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.963069 32551 raft_consensus.cc:729] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.963198 32551 consensus_queue.cc:260] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.963537 32551 ts_tablet_manager.cc:1398] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.963806 32551 tablet_bootstrap.cc:492] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:56:59.965590 32551 tablet_bootstrap.cc:654] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.966763 32551 tablet_bootstrap.cc:492] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:56:59.966886 32551 ts_tablet_manager.cc:1367] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:59.967450 32661 tablet_bootstrap.cc:492] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:59.967650 32550 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:59.967691 32662 tablet_bootstrap.cc:492] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:56:59.967746 32550 ts_tablet_manager.cc:1367] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:56:59.968147 32550 raft_consensus.cc:348] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:59.968241 32550 raft_consensus.cc:374] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.968268 32550 raft_consensus.cc:729] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.968374 32550 consensus_queue.cc:260] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:59.968675 32550 ts_tablet_manager.cc:1398] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.968900 32550 tablet_bootstrap.cc:492] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:56:59.968925 32551 raft_consensus.cc:348] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.969161 32551 raft_consensus.cc:374] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.969208 32551 raft_consensus.cc:729] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.969202 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:59.969334 32551 consensus_queue.cc:260] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.969803 32550 tablet_bootstrap.cc:654] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.969794 32551 ts_tablet_manager.cc:1398] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:59.973765 32661 tablet_bootstrap.cc:654] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.973963 32664 tablet_bootstrap.cc:492] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:56:59.975167 32664 tablet_bootstrap.cc:654] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.975353 32662 tablet_bootstrap.cc:654] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.978389 32647 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:59.978703 32647 tablet.cc:1620] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:56:59.978708 32662 tablet_bootstrap.cc:492] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:56:59.978816 32662 ts_tablet_manager.cc:1367] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:59.979355 32550 tablet_bootstrap.cc:492] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:56:59.979431 32550 ts_tablet_manager.cc:1367] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:59.979637 32649 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:59.979941 32649 tablet.cc:1620] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:56:59.979913 32550 raft_consensus.cc:348] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.980015 32550 raft_consensus.cc:374] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.980051 32550 raft_consensus.cc:729] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.980156 32550 consensus_queue.cc:260] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.980206 32648 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:59.980438 32550 ts_tablet_manager.cc:1398] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.980454 32648 tablet.cc:1620] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:56:59.981316 32662 raft_consensus.cc:348] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.981448 32662 raft_consensus.cc:374] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.981521 32662 raft_consensus.cc:729] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.983732 32662 consensus_queue.cc:260] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.984146 32662 ts_tablet_manager.cc:1398] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:59.984256 32661 tablet_bootstrap.cc:492] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:59.984390 32661 ts_tablet_manager.cc:1367] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:56:59.984632 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=ba6e5b25739548a1a13cd200757cecdb] alter complete (version 2)
I20240501 19:56:59.984894 32661 raft_consensus.cc:348] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.984999 32661 raft_consensus.cc:374] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.985172 32661 raft_consensus.cc:729] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.985637 32661 consensus_queue.cc:260] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.986091 32661 ts_tablet_manager.cc:1398] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:59.986326 32661 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:56:59.987401 32661 tablet_bootstrap.cc:654] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.995132 32664 tablet_bootstrap.cc:492] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:56:59.995257 32664 ts_tablet_manager.cc:1367] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 19:56:59.995836 32664 raft_consensus.cc:348] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.995941 32664 raft_consensus.cc:374] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.995976 32664 raft_consensus.cc:729] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.996109 32664 consensus_queue.cc:260] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:56:59.996410 32664 ts_tablet_manager.cc:1398] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:59.996866 32661 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:56:59.996994 32661 ts_tablet_manager.cc:1367] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:59.997535 32661 raft_consensus.cc:348] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:59.997818 32661 raft_consensus.cc:374] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.997879 32661 raft_consensus.cc:729] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.998072 32661 consensus_queue.cc:260] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:56:59.998621 32661 ts_tablet_manager.cc:1398] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:00.020346 32454 raft_consensus.cc:482] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.020552 32454 raft_consensus.cc:504] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.020972 32454 leader_election.cc:288] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:00.021507 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e20f102ed94db78eb4e735b843df24" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:00.021667 27276 raft_consensus.cc:2454] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:00.021606 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e20f102ed94db78eb4e735b843df24" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:00.021813 27424 raft_consensus.cc:2454] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:00.022186 26967 leader_election.cc:302] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:00.022471 32454 raft_consensus.cc:2781] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.022528 32454 raft_consensus.cc:482] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.022565 32454 raft_consensus.cc:3037] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.023607 32454 raft_consensus.cc:504] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.023979 32454 leader_election.cc:288] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:00.024305 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e20f102ed94db78eb4e735b843df24" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:00.024379 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e20f102ed94db78eb4e735b843df24" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:00.024435 27276 raft_consensus.cc:3037] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.024505 27424 raft_consensus.cc:3037] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.025437 27424 raft_consensus.cc:2454] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:00.025432 27276 raft_consensus.cc:2454] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:00.025844 26967 leader_election.cc:302] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:00.026018 32454 raft_consensus.cc:2781] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.026106 32454 raft_consensus.cc:686] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:00.026243 32454 consensus_queue.cc:237] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.028402 24452 catalog_manager.cc:5484] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:00.052659 31951 raft_consensus.cc:482] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.052784 31951 raft_consensus.cc:504] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:00.053228 31951 leader_election.cc:288] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:00.053917 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfa17e323af4bbcb123de23ad1102e7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:00.054061 27032 raft_consensus.cc:2454] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:00.054477 27227 leader_election.cc:302] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:00.054767 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfa17e323af4bbcb123de23ad1102e7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:00.054908 24594 raft_consensus.cc:2454] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:00.055979 31952 raft_consensus.cc:2781] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.056041 31952 raft_consensus.cc:482] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.056079 31952 raft_consensus.cc:3037] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.057264 31952 raft_consensus.cc:504] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:00.057602 31952 leader_election.cc:288] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:00.058962 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfa17e323af4bbcb123de23ad1102e7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:00.059731 27032 raft_consensus.cc:3037] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.060820 27032 raft_consensus.cc:2454] T abfa17e323af4bbcb123de23ad1102e7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:00.061152 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfa17e323af4bbcb123de23ad1102e7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:00.061272 24594 raft_consensus.cc:3037] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.062026 24594 raft_consensus.cc:2454] T abfa17e323af4bbcb123de23ad1102e7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:00.062269 27227 leader_election.cc:302] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:00.062448 31951 raft_consensus.cc:2781] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.062649 31951 raft_consensus.cc:686] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:00.062788 31951 consensus_queue.cc:237] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:00.065775 24448 catalog_manager.cc:5484] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:00.088312 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:00.093230 32643 raft_consensus.cc:482] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.093385 32643 raft_consensus.cc:504] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:00.093936 32643 leader_election.cc:288] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:00.094023 32647 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:00.094324 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37eadc7d034f11b68349f255227298" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:00.094453 32648 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:00.094503 27424 raft_consensus.cc:2454] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:00.095093 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37eadc7d034f11b68349f255227298" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:00.095247 27032 raft_consensus.cc:2454] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:00.095515 24791 leader_election.cc:302] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:00.095700 32643 raft_consensus.cc:2781] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.095759 32643 raft_consensus.cc:482] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.095804 32643 raft_consensus.cc:3037] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.096861 32649 tablet.cc:1722] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:00.097056 32643 raft_consensus.cc:504] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:00.097474 32643 leader_election.cc:288] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:00.098599 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37eadc7d034f11b68349f255227298" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:00.098735 27031 raft_consensus.cc:3037] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.099354 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37eadc7d034f11b68349f255227298" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:00.099460 27424 raft_consensus.cc:3037] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.099733 27031 raft_consensus.cc:2454] T 2e37eadc7d034f11b68349f255227298 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:00.100548 27424 raft_consensus.cc:2454] T 2e37eadc7d034f11b68349f255227298 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:00.101487 24793 leader_election.cc:302] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:00.102249 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=ba6e5b25739548a1a13cd200757cecdb] alter complete (version 3)
I20240501 19:57:00.102625 32643 raft_consensus.cc:2781] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.102731 32643 raft_consensus.cc:686] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:00.102869 32643 consensus_queue.cc:237] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:00.104992 24448 catalog_manager.cc:5484] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:00.130991 27011 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:00.131675 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:00.132979 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:00.133782 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:00.134958 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:00.146361 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:00.148857 24574 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:00.163200 27267 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:00.203150 24448 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:00.203315 24448 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:00.205560 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet ed08a34bb39741dcbeab8d2073e6fce1
I20240501 19:57:00.206663 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet ed08a34bb39741dcbeab8d2073e6fce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:00 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:00.207172 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet ed08a34bb39741dcbeab8d2073e6fce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:00 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:00.207325 32719 tablet_replica.cc:330] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:00.207445 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet ed08a34bb39741dcbeab8d2073e6fce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:00 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:00.207636 32591 tablet_replica.cc:330] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:00.207803 32591 raft_consensus.cc:2229] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:00.208030 32591 raft_consensus.cc:2258] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:00.208873 32591 ts_tablet_manager.cc:1875] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:00.210944 32720 tablet_replica.cc:330] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:00.211238 32720 raft_consensus.cc:2229] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:00.211465 32720 raft_consensus.cc:2258] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:00.211565 32591 ts_tablet_manager.cc:1888] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:00.211704 32591 log.cc:1196] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/ed08a34bb39741dcbeab8d2073e6fce1
I20240501 19:57:00.212018 32591 ts_tablet_manager.cc:1909] T ed08a34bb39741dcbeab8d2073e6fce1 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:57:00.212190 32720 ts_tablet_manager.cc:1875] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:00.212795 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet ed08a34bb39741dcbeab8d2073e6fce1 (table rebalancer_dynamic_table_new_name [id=ba6e5b25739548a1a13cd200757cecdb]) successfully deleted
I20240501 19:57:00.207509 32719 raft_consensus.cc:2229] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:00.213511 32719 raft_consensus.cc:2258] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:00.214149 32719 ts_tablet_manager.cc:1875] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:00.214576 32720 ts_tablet_manager.cc:1888] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:00.214704 32720 log.cc:1196] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/ed08a34bb39741dcbeab8d2073e6fce1
I20240501 19:57:00.215091 32720 ts_tablet_manager.cc:1909] T ed08a34bb39741dcbeab8d2073e6fce1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:57:00.215708 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet ed08a34bb39741dcbeab8d2073e6fce1 (table rebalancer_dynamic_table_new_name [id=ba6e5b25739548a1a13cd200757cecdb]) successfully deleted
I20240501 19:57:00.217159 32719 ts_tablet_manager.cc:1888] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:00.217274 32719 log.cc:1196] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/ed08a34bb39741dcbeab8d2073e6fce1
I20240501 19:57:00.217586 32719 ts_tablet_manager.cc:1909] T ed08a34bb39741dcbeab8d2073e6fce1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:00.218247 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet ed08a34bb39741dcbeab8d2073e6fce1 (table rebalancer_dynamic_table_new_name [id=ba6e5b25739548a1a13cd200757cecdb]) successfully deleted
I20240501 19:57:00.308792 24449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:00.316164 27011 tablet_service.cc:1467] Processing CreateTablet for tablet d79947a48a504626b7e54764cc1697ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]), partition=
I20240501 19:57:00.316336 24837 tablet_service.cc:1467] Processing CreateTablet for tablet d79947a48a504626b7e54764cc1697ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]), partition=
I20240501 19:57:00.316470 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79947a48a504626b7e54764cc1697ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.316561 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79947a48a504626b7e54764cc1697ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.316514 24574 tablet_service.cc:1467] Processing CreateTablet for tablet d79947a48a504626b7e54764cc1697ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]), partition=
I20240501 19:57:00.316805 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79947a48a504626b7e54764cc1697ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.318636 32664 tablet_bootstrap.cc:492] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:00.318730 32662 tablet_bootstrap.cc:492] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:00.318749 32551 tablet_bootstrap.cc:492] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:00.319640 32551 tablet_bootstrap.cc:654] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.319640 32664 tablet_bootstrap.cc:654] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.319639 32662 tablet_bootstrap.cc:654] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.320905 32664 tablet_bootstrap.cc:492] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:00.320987 32664 ts_tablet_manager.cc:1367] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:57:00.320989 32551 tablet_bootstrap.cc:492] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:00.321060 32551 ts_tablet_manager.cc:1367] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:00.321389 32551 raft_consensus.cc:348] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.321475 32551 raft_consensus.cc:374] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.321496 32551 raft_consensus.cc:729] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.321437 32664 raft_consensus.cc:348] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.321532 32664 raft_consensus.cc:374] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.321579 32664 raft_consensus.cc:729] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.321597 32551 consensus_queue.cc:260] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.321682 32664 consensus_queue.cc:260] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.321890 32551 ts_tablet_manager.cc:1398] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:00.321938 32664 ts_tablet_manager.cc:1398] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.322780 32662 tablet_bootstrap.cc:492] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:00.322872 32662 ts_tablet_manager.cc:1367] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:00.323402 32662 raft_consensus.cc:348] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.323511 32662 raft_consensus.cc:374] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.323544 32662 raft_consensus.cc:729] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.323661 32662 consensus_queue.cc:260] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.323935 32662 ts_tablet_manager.cc:1398] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.328742 24857 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:00.329341 32476 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:00.338117 27032 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:00.338570 32476 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:00.349354 24725 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:00.351272 32319 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:00.381287 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:00.388129 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:00.389703 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:00.393044 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:00.401741 27267 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:00.404289 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:00.410372 27011 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:00.436239 24574 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:00.489611 27276 consensus_queue.cc:237] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:00.490788 27162 raft_consensus.cc:1261] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:00.490788 24987 raft_consensus.cc:1261] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:00.491168 31952 consensus_queue.cc:1035] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:00.491350 31951 consensus_queue.cc:1035] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:00.493093 32705 raft_consensus.cc:2932] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.493577 24987 raft_consensus.cc:2932] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.494573 27162 raft_consensus.cc:2932] T 0ef4152fef8749f0994a952095225134 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.497330 24453 catalog_manager.cc:5484] T 0ef4152fef8749f0994a952095225134 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.500057 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef4152fef8749f0994a952095225134 with delete_type TABLET_DATA_TOMBSTONED (TS 4a6e54e86a9949e5970b42e8e32e7561 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:00.500465 393 tablet_replica.cc:330] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:00.500684 393 raft_consensus.cc:2229] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:00.500828 393 raft_consensus.cc:2258] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:00.501278 393 ts_tablet_manager.cc:1875] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:00.503614 393 ts_tablet_manager.cc:1888] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:00.503702 393 log.cc:1196] T 0ef4152fef8749f0994a952095225134 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/0ef4152fef8749f0994a952095225134
I20240501 19:57:00.504280 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 0ef4152fef8749f0994a952095225134 (table rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]) successfully deleted
I20240501 19:57:00.526834 27424 consensus_queue.cc:237] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.528427 27276 raft_consensus.cc:1261] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.528463 27032 raft_consensus.cc:1261] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.528784 30935 consensus_queue.cc:1035] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:00.528939 27359 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fce9f9c2a5be47d3986e3d7b53570b12. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.529090 416 consensus_queue.cc:1035] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:00.530987 30935 raft_consensus.cc:2932] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.531431 27032 raft_consensus.cc:2932] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.534547 24449 catalog_manager.cc:5484] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.535166 32643 consensus_queue.cc:1035] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.538311 27276 raft_consensus.cc:2932] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.545457 32643 consensus_queue.cc:1035] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.564167 24856 consensus_queue.cc:237] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.565893 24725 raft_consensus.cc:1261] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:00.566191 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfb037c9bf1a4ed080745df687ac4a89. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.566654 24594 raft_consensus.cc:1261] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:00.566860 31550 consensus_queue.cc:1035] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:00.567564 440 consensus_queue.cc:1035] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:00.567564 32454 consensus_queue.cc:1035] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.571015 31550 raft_consensus.cc:2932] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:00.571864 30967 consensus_queue.cc:1035] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.574815 24594 raft_consensus.cc:2932] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:00.575400 24453 catalog_manager.cc:5484] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:00.578109 24725 raft_consensus.cc:2932] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:00.611420 24987 consensus_queue.cc:237] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
W20240501 19:57:00.612890 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53981ad3756d4c54994f7aea973c5cde. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.612932 24856 raft_consensus.cc:1261] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:00.613328 463 consensus_queue.cc:1035] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:00.613724 24594 raft_consensus.cc:1261] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:00.614800 463 consensus_queue.cc:1035] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:00.616752 463 raft_consensus.cc:2932] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.617529 24594 raft_consensus.cc:2932] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.620251 24453 catalog_manager.cc:5484] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.622159 24856 raft_consensus.cc:2932] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.625605 24449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:00.629045 32705 consensus_queue.cc:1035] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.634543 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 2a075e1cbd5d462b8f140c7e01c8ea9f (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.634681 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 2a075e1cbd5d462b8f140c7e01c8ea9f (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.634869 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a075e1cbd5d462b8f140c7e01c8ea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.634927 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a075e1cbd5d462b8f140c7e01c8ea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.635141 32705 consensus_queue.cc:1035] T abfa17e323af4bbcb123de23ad1102e7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.637730 24574 tablet_service.cc:1467] Processing CreateTablet for tablet dd1df8a61b1c41368727514e30e263a8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.638018 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1df8a61b1c41368727514e30e263a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.638379 475 tablet_bootstrap.cc:492] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:00.638474 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 7eaf6cd60c754ba99747df02a43b1b15 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.638617 24705 tablet_service.cc:1467] Processing CreateTablet for tablet dd1df8a61b1c41368727514e30e263a8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.638723 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eaf6cd60c754ba99747df02a43b1b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.638887 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1df8a61b1c41368727514e30e263a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.639405 475 tablet_bootstrap.cc:654] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.640174 27273 tablet_service.cc:1467] Processing CreateTablet for tablet dd1df8a61b1c41368727514e30e263a8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.640450 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1df8a61b1c41368727514e30e263a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.640861 32664 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:00.642954 477 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:00.643124 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 7eaf6cd60c754ba99747df02a43b1b15 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.643338 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eaf6cd60c754ba99747df02a43b1b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.644102 477 tablet_bootstrap.cc:654] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.645152 477 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:00.645227 477 ts_tablet_manager.cc:1367] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:00.645661 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 2a075e1cbd5d462b8f140c7e01c8ea9f (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.645702 477 raft_consensus.cc:348] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.645802 477 raft_consensus.cc:374] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.645840 477 raft_consensus.cc:729] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.645885 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a075e1cbd5d462b8f140c7e01c8ea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.645956 477 consensus_queue.cc:260] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.646230 477 ts_tablet_manager.cc:1398] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.646696 32664 tablet_bootstrap.cc:654] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.647575 477 tablet_bootstrap.cc:492] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:00.648236 474 tablet_bootstrap.cc:492] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:00.649155 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 7eaf6cd60c754ba99747df02a43b1b15 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.649360 474 tablet_bootstrap.cc:654] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.650831 474 tablet_bootstrap.cc:492] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:00.650920 474 ts_tablet_manager.cc:1367] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:57:00.651166 32664 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:00.651248 32664 ts_tablet_manager.cc:1367] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 19:57:00.651696 32664 raft_consensus.cc:348] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.651803 32664 raft_consensus.cc:374] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.651837 32664 raft_consensus.cc:729] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.651942 32664 consensus_queue.cc:260] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.652213 32664 ts_tablet_manager.cc:1398] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:00.652658 477 tablet_bootstrap.cc:654] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.654594 478 tablet_bootstrap.cc:492] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:00.655642 478 tablet_bootstrap.cc:654] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.651849 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eaf6cd60c754ba99747df02a43b1b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.657987 475 tablet_bootstrap.cc:492] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:00.658087 475 ts_tablet_manager.cc:1367] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:57:00.658666 475 raft_consensus.cc:348] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.658784 475 raft_consensus.cc:374] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.658818 475 raft_consensus.cc:729] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.658933 475 consensus_queue.cc:260] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.659260 475 ts_tablet_manager.cc:1398] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:00.659850 30967 raft_consensus.cc:482] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.659943 30967 raft_consensus.cc:504] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.660418 30967 leader_election.cc:288] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:00.660355 474 raft_consensus.cc:348] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.660477 474 raft_consensus.cc:374] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.660514 474 raft_consensus.cc:729] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.661156 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79947a48a504626b7e54764cc1697ac" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:00.661291 24594 raft_consensus.cc:2454] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:00.661337 474 consensus_queue.cc:260] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.661635 474 ts_tablet_manager.cc:1398] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:00.661607 26965 leader_election.cc:302] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:00.661765 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79947a48a504626b7e54764cc1697ac" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:00.661895 24856 raft_consensus.cc:2454] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:00.662422 30967 raft_consensus.cc:2781] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.662495 30967 raft_consensus.cc:482] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.661911 474 tablet_bootstrap.cc:492] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:00.662542 30967 raft_consensus.cc:3037] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.663630 30967 raft_consensus.cc:504] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.663981 30967 leader_election.cc:288] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:00.665225 474 tablet_bootstrap.cc:654] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.665409 478 tablet_bootstrap.cc:492] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:00.665424 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79947a48a504626b7e54764cc1697ac" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:00.665467 487 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:00.665534 24594 raft_consensus.cc:3037] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.665561 478 ts_tablet_manager.cc:1367] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240501 19:57:00.666010 478 raft_consensus.cc:348] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.666106 478 raft_consensus.cc:374] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.666136 478 raft_consensus.cc:729] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.666251 478 consensus_queue.cc:260] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.666579 24594 raft_consensus.cc:2454] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:00.667268 26965 leader_election.cc:302] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:00.667531 30967 raft_consensus.cc:2781] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.667528 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79947a48a504626b7e54764cc1697ac" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:00.667609 30967 raft_consensus.cc:686] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:00.667646 24856 raft_consensus.cc:3037] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.667735 30967 consensus_queue.cc:237] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.668661 24856 raft_consensus.cc:2454] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:00.669498 478 ts_tablet_manager.cc:1398] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:57:00.670562 24453 catalog_manager.cc:5484] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.669620 487 tablet_bootstrap.cc:654] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.673298 474 tablet_bootstrap.cc:492] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:00.673429 474 ts_tablet_manager.cc:1367] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:00.673868 474 raft_consensus.cc:348] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.673974 474 raft_consensus.cc:374] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.674293 474 raft_consensus.cc:729] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.674507 474 consensus_queue.cc:260] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.674834 474 ts_tablet_manager.cc:1398] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.676380 477 tablet_bootstrap.cc:492] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:00.675230 487 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:00.676491 477 ts_tablet_manager.cc:1367] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 19:57:00.676561 487 ts_tablet_manager.cc:1367] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:00.676918 487 raft_consensus.cc:348] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.677021 487 raft_consensus.cc:374] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.677044 487 raft_consensus.cc:729] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.676999 477 raft_consensus.cc:348] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.677104 477 raft_consensus.cc:374] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.677135 477 raft_consensus.cc:729] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.677147 487 consensus_queue.cc:260] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.677248 477 consensus_queue.cc:260] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.677500 487 ts_tablet_manager.cc:1398] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.677539 477 ts_tablet_manager.cc:1398] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.677708 477 tablet_bootstrap.cc:492] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:00.678040 355 raft_consensus.cc:482] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.678121 355 raft_consensus.cc:504] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.678541 355 leader_election.cc:288] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:00.678580 477 tablet_bootstrap.cc:654] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.679147 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1df8a61b1c41368727514e30e263a8" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:00.679258 27276 raft_consensus.cc:2454] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:00.679543 24531 leader_election.cc:302] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:00.679694 355 raft_consensus.cc:2781] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.679749 355 raft_consensus.cc:482] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.679795 355 raft_consensus.cc:3037] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.680812 355 raft_consensus.cc:504] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.681125 355 leader_election.cc:288] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:00.681324 477 tablet_bootstrap.cc:492] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:00.681396 477 ts_tablet_manager.cc:1367] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:00.681404 24856 consensus_queue.cc:237] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:00.681731 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1df8a61b1c41368727514e30e263a8" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:00.681818 27276 raft_consensus.cc:3037] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.681854 477 raft_consensus.cc:348] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.681944 477 raft_consensus.cc:374] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.681975 477 raft_consensus.cc:729] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.682096 477 consensus_queue.cc:260] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.682384 477 ts_tablet_manager.cc:1398] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:00.682597 27276 raft_consensus.cc:2454] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:00.683135 24531 leader_election.cc:302] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:00.683179 27276 raft_consensus.cc:1261] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.683192 27163 raft_consensus.cc:1261] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.683617 355 raft_consensus.cc:2781] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.683701 355 raft_consensus.cc:686] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:00.683827 355 consensus_queue.cc:237] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.683846 31550 consensus_queue.cc:1035] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:00.684295 24792 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c52abe3cfff44f88f41e1e885e5f2c9. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.684561 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1df8a61b1c41368727514e30e263a8" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:00.684671 24725 raft_consensus.cc:2454] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:00.684824 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1df8a61b1c41368727514e30e263a8" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:00.684900 24727 raft_consensus.cc:3037] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.685858 24727 raft_consensus.cc:2454] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:00.686046 24449 catalog_manager.cc:5484] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.686146 31550 consensus_queue.cc:1035] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:00.688249 31550 raft_consensus.cc:2932] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:00.691318 27163 raft_consensus.cc:2932] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:00.691457 27276 raft_consensus.cc:2932] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:00.694126 24449 catalog_manager.cc:5484] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:00.711241 24856 consensus_queue.cc:237] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:00.712641 24594 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:00.712659 27163 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:57:00.712909 24791 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d2603a318c34256a9fa28979ae41fec. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.713109 32643 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:00.713270 31550 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:00.715023 32643 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:00.715401 27163 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:00.717571 24594 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:00.718416 24451 catalog_manager.cc:5484] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:00.737639 24594 consensus_queue.cc:237] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:00.739143 24856 raft_consensus.cc:1261] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:00.739182 24987 raft_consensus.cc:1261] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:00.739457 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a37dfc7a4324e2b8c43acb51d07a277. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.739531 32476 consensus_queue.cc:1035] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:00.739706 32319 consensus_queue.cc:1035] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:00.741624 32476 raft_consensus.cc:2932] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.742003 24856 raft_consensus.cc:2932] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.743549 24987 raft_consensus.cc:2932] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.744503 24451 catalog_manager.cc:5484] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.762614 32705 raft_consensus.cc:482] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.762727 32705 raft_consensus.cc:504] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.763100 32705 leader_election.cc:288] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:00.763502 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eaf6cd60c754ba99747df02a43b1b15" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:00.763645 27163 raft_consensus.cc:2454] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:00.763918 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eaf6cd60c754ba99747df02a43b1b15" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:00.763947 27229 leader_election.cc:302] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:00.764046 27424 raft_consensus.cc:2454] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:00.764106 32705 raft_consensus.cc:2781] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.764154 32705 raft_consensus.cc:482] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.764196 32705 raft_consensus.cc:3037] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.765334 32705 raft_consensus.cc:504] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.765702 32705 leader_election.cc:288] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:00.766536 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eaf6cd60c754ba99747df02a43b1b15" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:00.766638 27163 raft_consensus.cc:3037] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.766621 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eaf6cd60c754ba99747df02a43b1b15" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:00.766733 27424 raft_consensus.cc:3037] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.767630 27163 raft_consensus.cc:2454] T 7eaf6cd60c754ba99747df02a43b1b15 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:00.767712 27424 raft_consensus.cc:2454] T 7eaf6cd60c754ba99747df02a43b1b15 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:00.768105 27229 leader_election.cc:302] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:00.768445 32705 raft_consensus.cc:2781] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.768517 32705 raft_consensus.cc:686] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:00.768623 32705 consensus_queue.cc:237] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:00.770725 24449 catalog_manager.cc:5484] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:00.772987 27163 consensus_queue.cc:237] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:00.774405 27276 raft_consensus.cc:1261] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:00.774552 27097 consensus_peers.cc:485] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92906e09f3474d7987082e0d33304886. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.774999 490 consensus_queue.cc:1035] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:00.775332 27424 raft_consensus.cc:1261] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.775738 490 consensus_queue.cc:1035] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:00.777863 490 raft_consensus.cc:2932] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:00.778116 27424 raft_consensus.cc:2932] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:00.779920 27276 raft_consensus.cc:2932] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:00.783725 24449 catalog_manager.cc:5484] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:00.798175 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:00.802143 24594 raft_consensus.cc:1261] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.802217 24856 raft_consensus.cc:1261] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.802564 30967 consensus_queue.cc:1035] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.802716 32454 consensus_queue.cc:1035] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.805768 579 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:00.806382 579 tablet.cc:1620] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:00.806804 24449 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet d79947a48a504626b7e54764cc1697ac (table rebalancer_dynamic_table [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]). Expected version 1 got 0
I20240501 19:57:00.807173 24453 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet d79947a48a504626b7e54764cc1697ac (table rebalancer_dynamic_table [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]). Expected version 1 got 0
I20240501 19:57:00.807360 580 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:00.807655 580 tablet.cc:1620] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:00.808394 581 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:00.808737 27032 consensus_queue.cc:237] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.809392 24453 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet d79947a48a504626b7e54764cc1697ac (table rebalancer_dynamic_table [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]). Expected version 1 got 0
I20240501 19:57:00.810153 27276 raft_consensus.cc:1261] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.810163 27424 raft_consensus.cc:1261] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.810576 30967 consensus_queue.cc:1035] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:00.811013 581 tablet.cc:1620] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:00.811373 30967 consensus_queue.cc:1035] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:00.811980 26965 consensus_peers.cc:485] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8eb1e61bb542415caac5978943e6b3eb. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.813251 30967 raft_consensus.cc:2932] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.813623 27424 raft_consensus.cc:2932] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.815830 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=55fd4acfd1d24dc7bbec0b91da1ffe2d] alter complete (version 1)
I20240501 19:57:00.817422 24453 catalog_manager.cc:5484] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.833235 27276 raft_consensus.cc:2932] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.851480 24594 consensus_queue.cc:237] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:00.852896 24856 raft_consensus.cc:1261] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:00.853037 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0064e3e3ae1548bba511bd76fb4ec3e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.853070 24987 raft_consensus.cc:1261] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:00.853614 355 consensus_queue.cc:1035] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:00.853811 32476 consensus_queue.cc:1035] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:00.855490 355 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.855916 24987 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.857712 24856 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.858839 24449 catalog_manager.cc:5484] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:00.877492 27424 consensus_queue.cc:237] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.878855 27032 raft_consensus.cc:1261] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.878857 24987 raft_consensus.cc:1261] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:00.878890 27359 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fed0e67f44543d09dbeda7100c7e0d6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:00.879313 419 consensus_queue.cc:1035] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:00.879498 416 consensus_queue.cc:1035] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:00.881420 416 raft_consensus.cc:2932] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.881783 27032 raft_consensus.cc:2932] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.883131 24987 raft_consensus.cc:2932] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.884505 24453 catalog_manager.cc:5484] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:00.921217 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:00.926522 648 ts_tablet_manager.cc:902] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:00.927243 648 tablet_copy_client.cc:308] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.8:35957
I20240501 19:57:00.928264 27431 tablet_copy_service.cc:140] P 4a6e54e86a9949e5970b42e8e32e7561: Received BeginTabletCopySession request for tablet fce9f9c2a5be47d3986e3d7b53570b12 from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:56347)
I20240501 19:57:00.928408 27431 tablet_copy_service.cc:161] P 4a6e54e86a9949e5970b42e8e32e7561: Beginning new tablet copy session on tablet fce9f9c2a5be47d3986e3d7b53570b12 from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:56347: session id = bbf7735f82ca4f65a5170ac7909faa47-fce9f9c2a5be47d3986e3d7b53570b12
I20240501 19:57:00.929638 579 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:00.929720 27431 tablet_copy_source_session.cc:205] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:00.929924 579 tablet.cc:1620] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:00.930891 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce9f9c2a5be47d3986e3d7b53570b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.933269 648 tablet_copy_client.cc:783] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:00.933487 581 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:00.934386 581 tablet.cc:1620] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:00.934379 580 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:00.934655 580 tablet.cc:1620] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:00.933439 648 tablet_copy_client.cc:647] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:00.941334 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=55fd4acfd1d24dc7bbec0b91da1ffe2d] alter complete (version 2)
I20240501 19:57:00.946246 648 tablet_copy_client.cc:523] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:00.947762 648 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:00.969127 31952 raft_consensus.cc:482] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.969244 31952 raft_consensus.cc:504] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.969570 31952 leader_election.cc:288] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:00.969883 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a075e1cbd5d462b8f140c7e01c8ea9f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:00.970016 24987 raft_consensus.cc:2454] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:00.970023 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a075e1cbd5d462b8f140c7e01c8ea9f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:00.970161 27163 raft_consensus.cc:2454] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:00.970446 27228 leader_election.cc:302] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:00.970799 31952 raft_consensus.cc:2781] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.970855 31952 raft_consensus.cc:482] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.970891 31952 raft_consensus.cc:3037] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.972007 31952 raft_consensus.cc:504] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.972376 31952 leader_election.cc:288] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:00.972908 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a075e1cbd5d462b8f140c7e01c8ea9f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:00.973004 24987 raft_consensus.cc:3037] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.973889 24987 raft_consensus.cc:2454] T 2a075e1cbd5d462b8f140c7e01c8ea9f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:00.974169 27228 leader_election.cc:302] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:00.974256 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a075e1cbd5d462b8f140c7e01c8ea9f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:00.974375 27163 raft_consensus.cc:3037] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.974558 31952 raft_consensus.cc:2781] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.974650 31952 raft_consensus.cc:686] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:00.974772 31952 consensus_queue.cc:237] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:00.975484 27163 raft_consensus.cc:2454] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:00.977685 24451 catalog_manager.cc:5484] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:00.980717 648 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:00.980989 648 tablet_bootstrap.cc:492] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:00.981156 648 ts_tablet_manager.cc:1367] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20240501 19:57:00.981613 648 raft_consensus.cc:348] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.981716 648 raft_consensus.cc:729] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.981979 648 consensus_queue.cc:260] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:00.982338 648 ts_tablet_manager.cc:1398] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:57:00.982825 27431 tablet_copy_service.cc:342] P 4a6e54e86a9949e5970b42e8e32e7561: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-fce9f9c2a5be47d3986e3d7b53570b12 received from {username='slave'} at 127.17.65.1:56347
I20240501 19:57:00.982915 27431 tablet_copy_service.cc:433] P 4a6e54e86a9949e5970b42e8e32e7561: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-fce9f9c2a5be47d3986e3d7b53570b12 on tablet fce9f9c2a5be47d3986e3d7b53570b12 with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:01.003042 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:01.010890 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:01.017544 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:01.019456 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:01.021265 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:01.022219 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:01.036695 687 ts_tablet_manager.cc:902] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:01.039502 687 tablet_copy_client.cc:308] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:57:01.040087 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet cfb037c9bf1a4ed080745df687ac4a89 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:39423)
I20240501 19:57:01.040199 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet cfb037c9bf1a4ed080745df687ac4a89 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:39423: session id = 4a6e54e86a9949e5970b42e8e32e7561-cfb037c9bf1a4ed080745df687ac4a89
I20240501 19:57:01.041090 24878 tablet_copy_source_session.cc:205] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.042049 687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb037c9bf1a4ed080745df687ac4a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.044705 687 tablet_copy_client.cc:783] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.044910 687 tablet_copy_client.cc:647] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.045560 24446 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:01.051177 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:01.053388 687 tablet_copy_client.cc:523] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.056085 579 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:01.056289 687 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:01.057312 580 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:01.059051 581 tablet.cc:1722] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:01.065894 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=55fd4acfd1d24dc7bbec0b91da1ffe2d] alter complete (version 3)
I20240501 19:57:01.081024 27267 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:01.085776 687 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.086102 687 tablet_bootstrap.cc:492] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:57:01.086272 687 ts_tablet_manager.cc:1367] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.004s
I20240501 19:57:01.086730 687 raft_consensus.cc:348] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:01.086854 687 raft_consensus.cc:729] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.086974 687 consensus_queue.cc:260] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:01.087327 687 ts_tablet_manager.cc:1398] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:57:01.087975 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-cfb037c9bf1a4ed080745df687ac4a89 received from {username='slave'} at 127.17.65.8:39423
I20240501 19:57:01.088057 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-cfb037c9bf1a4ed080745df687ac4a89 on tablet cfb037c9bf1a4ed080745df687ac4a89 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:01.099645 686 ts_tablet_manager.cc:902] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:01.100970 686 tablet_copy_client.cc:308] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:57:01.102197 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet 53981ad3756d4c54994f7aea973c5cde from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:51823)
I20240501 19:57:01.102310 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet 53981ad3756d4c54994f7aea973c5cde from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:51823: session id = 29d1a658e2b946e0b412e5fef82db9d9-53981ad3756d4c54994f7aea973c5cde
I20240501 19:57:01.103458 25009 tablet_copy_source_session.cc:205] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.103961 32476 consensus_queue.cc:1035] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.106684 686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53981ad3756d4c54994f7aea973c5cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.107332 32319 consensus_queue.cc:1035] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.112744 686 tablet_copy_client.cc:783] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.113102 686 tablet_copy_client.cc:647] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.114564 686 tablet_copy_client.cc:523] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.115880 686 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:01.128747 712 ts_tablet_manager.cc:902] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:01.129474 712 tablet_copy_client.cc:308] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:57:01.130368 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 5a37dfc7a4324e2b8c43acb51d07a277 from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:53099)
I20240501 19:57:01.130487 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 5a37dfc7a4324e2b8c43acb51d07a277 from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:53099: session id = 29d1a658e2b946e0b412e5fef82db9d9-5a37dfc7a4324e2b8c43acb51d07a277
I20240501 19:57:01.131464 24616 tablet_copy_source_session.cc:205] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.132272 712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a37dfc7a4324e2b8c43acb51d07a277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.134167 712 tablet_copy_client.cc:783] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.134322 712 tablet_copy_client.cc:647] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.135684 712 tablet_copy_client.cc:523] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.136927 712 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:01.139699 686 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.139950 686 tablet_bootstrap.cc:492] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:57:01.140105 686 ts_tablet_manager.cc:1367] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.003s
I20240501 19:57:01.140532 686 raft_consensus.cc:348] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:01.140630 686 raft_consensus.cc:729] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.140784 686 consensus_queue.cc:260] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:01.141036 686 ts_tablet_manager.cc:1398] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.143369 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-53981ad3756d4c54994f7aea973c5cde received from {username='slave'} at 127.17.65.6:51823
I20240501 19:57:01.143472 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-53981ad3756d4c54994f7aea973c5cde on tablet 53981ad3756d4c54994f7aea973c5cde with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:01.169009 712 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.169358 712 tablet_bootstrap.cc:492] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:57:01.169627 712 ts_tablet_manager.cc:1367] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.000s
I20240501 19:57:01.169843 24448 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:01.169982 24448 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:01.170109 712 raft_consensus.cc:348] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:01.170212 712 raft_consensus.cc:729] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.170351 712 consensus_queue.cc:260] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:01.171337 712 ts_tablet_manager.cc:1398] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.172895 24448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet d79947a48a504626b7e54764cc1697ac
I20240501 19:57:01.173511 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet d79947a48a504626b7e54764cc1697ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:01 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:01.173617 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet d79947a48a504626b7e54764cc1697ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:01 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:01.173830 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet d79947a48a504626b7e54764cc1697ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:01 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:01.173933 729 tablet_replica.cc:330] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:01.174155 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-5a37dfc7a4324e2b8c43acb51d07a277 received from {username='slave'} at 127.17.65.6:53099
I20240501 19:57:01.174196 729 raft_consensus.cc:2229] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:01.174247 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-5a37dfc7a4324e2b8c43acb51d07a277 on tablet 5a37dfc7a4324e2b8c43acb51d07a277 with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:01.174372 728 tablet_replica.cc:330] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:01.174412 729 raft_consensus.cc:2258] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:01.174532 728 raft_consensus.cc:2229] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:01.174743 728 raft_consensus.cc:2258] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:01.175473 729 ts_tablet_manager.cc:1875] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:01.177910 729 ts_tablet_manager.cc:1888] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:01.178018 729 log.cc:1196] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/d79947a48a504626b7e54764cc1697ac
I20240501 19:57:01.178359 729 ts_tablet_manager.cc:1909] T d79947a48a504626b7e54764cc1697ac P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:57:01.178761 728 ts_tablet_manager.cc:1875] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:01.179224 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet d79947a48a504626b7e54764cc1697ac (table rebalancer_dynamic_table_new_name [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]) successfully deleted
I20240501 19:57:01.181859 728 ts_tablet_manager.cc:1888] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:01.182008 728 log.cc:1196] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/d79947a48a504626b7e54764cc1697ac
I20240501 19:57:01.182356 728 ts_tablet_manager.cc:1909] T d79947a48a504626b7e54764cc1697ac P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:01.183255 730 tablet_replica.cc:330] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:01.183418 730 raft_consensus.cc:2229] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:01.183542 730 raft_consensus.cc:2258] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:01.184154 730 ts_tablet_manager.cc:1875] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:01.184727 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet d79947a48a504626b7e54764cc1697ac (table rebalancer_dynamic_table_new_name [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]) successfully deleted
I20240501 19:57:01.185056 752 ts_tablet_manager.cc:902] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:01.185614 752 tablet_copy_client.cc:308] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:57:01.186085 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 6d2603a318c34256a9fa28979ae41fec from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:53857)
I20240501 19:57:01.186190 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 6d2603a318c34256a9fa28979ae41fec from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:53857: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-6d2603a318c34256a9fa28979ae41fec
I20240501 19:57:01.186663 730 ts_tablet_manager.cc:1888] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:01.186733 730 log.cc:1196] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/d79947a48a504626b7e54764cc1697ac
I20240501 19:57:01.187053 730 ts_tablet_manager.cc:1909] T d79947a48a504626b7e54764cc1697ac P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:57:01.187286 24878 tablet_copy_source_session.cc:205] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.187695 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet d79947a48a504626b7e54764cc1697ac (table rebalancer_dynamic_table_new_name [id=55fd4acfd1d24dc7bbec0b91da1ffe2d]) successfully deleted
I20240501 19:57:01.189361 752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2603a318c34256a9fa28979ae41fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.194825 752 tablet_copy_client.cc:783] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.198670 752 tablet_copy_client.cc:647] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.199944 752 tablet_copy_client.cc:523] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.201730 752 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:01.235942 712 ts_tablet_manager.cc:902] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:01.236439 712 tablet_copy_client.cc:308] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:57:01.236966 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 0064e3e3ae1548bba511bd76fb4ec3e1 from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:53099)
I20240501 19:57:01.237080 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 0064e3e3ae1548bba511bd76fb4ec3e1 from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:53099: session id = 29d1a658e2b946e0b412e5fef82db9d9-0064e3e3ae1548bba511bd76fb4ec3e1
I20240501 19:57:01.238114 24616 tablet_copy_source_session.cc:205] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.238122 752 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.238476 752 tablet_bootstrap.cc:492] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:57:01.238742 752 ts_tablet_manager.cc:1367] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.000s
I20240501 19:57:01.238965 712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0064e3e3ae1548bba511bd76fb4ec3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.239377 752 raft_consensus.cc:348] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:01.239547 752 raft_consensus.cc:729] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.239748 752 consensus_queue.cc:260] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:01.241127 712 tablet_copy_client.cc:783] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.241286 712 tablet_copy_client.cc:647] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.241361 752 ts_tablet_manager.cc:1398] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.241922 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-6d2603a318c34256a9fa28979ae41fec received from {username='slave'} at 127.17.65.5:53857
I20240501 19:57:01.242008 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-6d2603a318c34256a9fa28979ae41fec on tablet 6d2603a318c34256a9fa28979ae41fec with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:57:01.246731 712 tablet_copy_client.cc:523] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.248009 712 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:01.264801 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:01.267899 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:01.267927 32705 consensus_queue.cc:1035] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.271960 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:01.273370 27267 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:01.279210 24446 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:01.285955 27267 tablet_service.cc:1467] Processing CreateTablet for tablet cbecd795f2f4471ab387b528a8b1aea3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029]), partition=
I20240501 19:57:01.286201 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbecd795f2f4471ab387b528a8b1aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.286849 27011 tablet_service.cc:1467] Processing CreateTablet for tablet cbecd795f2f4471ab387b528a8b1aea3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029]), partition=
I20240501 19:57:01.287120 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbecd795f2f4471ab387b528a8b1aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.287712 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:01.290107 24837 tablet_service.cc:1467] Processing CreateTablet for tablet cbecd795f2f4471ab387b528a8b1aea3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029]), partition=
I20240501 19:57:01.290333 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbecd795f2f4471ab387b528a8b1aea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.295368 712 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.295311 31952 consensus_queue.cc:1035] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.295665 712 tablet_bootstrap.cc:492] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:57:01.295862 712 ts_tablet_manager.cc:1367] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.048s user 0.025s sys 0.000s
I20240501 19:57:01.296345 712 raft_consensus.cc:348] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:01.296478 712 raft_consensus.cc:729] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.296655 712 consensus_queue.cc:260] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:01.297948 712 ts_tablet_manager.cc:1398] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:01.300930 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:01.303213 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-0064e3e3ae1548bba511bd76fb4ec3e1 received from {username='slave'} at 127.17.65.6:53099
I20240501 19:57:01.303287 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-0064e3e3ae1548bba511bd76fb4ec3e1 on tablet 0064e3e3ae1548bba511bd76fb4ec3e1 with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:01.304935 27011 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:01.311686 777 tablet_bootstrap.cc:492] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:01.311206 776 tablet_bootstrap.cc:492] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:01.313274 777 tablet_bootstrap.cc:654] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.315656 776 tablet_bootstrap.cc:654] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.316475 782 ts_tablet_manager.cc:902] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:01.317191 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:01.317220 782 tablet_copy_client.cc:308] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:57:01.317754 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 2c52abe3cfff44f88f41e1e885e5f2c9 from peer b9fd7c49d12b48a7ae73ab09876a0b82 ({username='slave'} at 127.17.65.4:41261)
I20240501 19:57:01.317840 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 2c52abe3cfff44f88f41e1e885e5f2c9 from peer b9fd7c49d12b48a7ae73ab09876a0b82 at {username='slave'} at 127.17.65.4:41261: session id = b9fd7c49d12b48a7ae73ab09876a0b82-2c52abe3cfff44f88f41e1e885e5f2c9
I20240501 19:57:01.318662 24878 tablet_copy_source_session.cc:205] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.320292 782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c52abe3cfff44f88f41e1e885e5f2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.322701 775 ts_tablet_manager.cc:902] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:01.325065 775 tablet_copy_client.cc:308] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.6:44579
I20240501 19:57:01.327818 27183 tablet_copy_service.cc:140] P 29d1a658e2b946e0b412e5fef82db9d9: Received BeginTabletCopySession request for tablet 92906e09f3474d7987082e0d33304886 from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:50333)
I20240501 19:57:01.327926 27183 tablet_copy_service.cc:161] P 29d1a658e2b946e0b412e5fef82db9d9: Beginning new tablet copy session on tablet 92906e09f3474d7987082e0d33304886 from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:50333: session id = 9d64cdfaeb2d46b29845160e395fc241-92906e09f3474d7987082e0d33304886
I20240501 19:57:01.328866 27183 tablet_copy_source_session.cc:205] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.330070 775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92906e09f3474d7987082e0d33304886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.335238 775 tablet_copy_client.cc:783] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.335489 775 tablet_copy_client.cc:647] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.339887 776 tablet_bootstrap.cc:492] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:01.340395 777 tablet_bootstrap.cc:492] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:01.340562 777 ts_tablet_manager.cc:1367] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 19:57:01.341059 782 tablet_copy_client.cc:783] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.341221 782 tablet_copy_client.cc:647] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.341233 776 ts_tablet_manager.cc:1367] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240501 19:57:01.341775 777 raft_consensus.cc:348] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.342167 776 raft_consensus.cc:348] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.342347 777 raft_consensus.cc:374] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.342347 782 tablet_copy_client.cc:523] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.342460 777 raft_consensus.cc:729] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.342725 777 consensus_queue.cc:260] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.342949 776 raft_consensus.cc:374] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.343108 776 raft_consensus.cc:729] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.343467 777 ts_tablet_manager.cc:1398] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:01.343914 776 consensus_queue.cc:260] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.344785 782 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:01.345177 776 ts_tablet_manager.cc:1398] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:01.351634 648 ts_tablet_manager.cc:902] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:01.352093 648 tablet_copy_client.cc:308] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.8:35957
I20240501 19:57:01.353134 27431 tablet_copy_service.cc:140] P 4a6e54e86a9949e5970b42e8e32e7561: Received BeginTabletCopySession request for tablet 2fed0e67f44543d09dbeda7100c7e0d6 from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:56347)
I20240501 19:57:01.353246 27431 tablet_copy_service.cc:161] P 4a6e54e86a9949e5970b42e8e32e7561: Beginning new tablet copy session on tablet 2fed0e67f44543d09dbeda7100c7e0d6 from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:56347: session id = bbf7735f82ca4f65a5170ac7909faa47-2fed0e67f44543d09dbeda7100c7e0d6
I20240501 19:57:01.353528 782 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.353762 782 tablet_bootstrap.cc:492] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap complete.
I20240501 19:57:01.353911 782 ts_tablet_manager.cc:1367] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240501 19:57:01.354394 27431 tablet_copy_source_session.cc:205] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.354375 782 raft_consensus.cc:348] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:01.354491 782 raft_consensus.cc:729] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.354642 782 consensus_queue.cc:260] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:01.355000 782 ts_tablet_manager.cc:1398] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 19:57:01.355448 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed0e67f44543d09dbeda7100c7e0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.355870 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-2c52abe3cfff44f88f41e1e885e5f2c9 received from {username='slave'} at 127.17.65.4:41261
I20240501 19:57:01.355968 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-2c52abe3cfff44f88f41e1e885e5f2c9 on tablet 2c52abe3cfff44f88f41e1e885e5f2c9 with peer b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:01.357774 648 tablet_copy_client.cc:783] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.357915 648 tablet_copy_client.cc:647] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.363508 783 tablet_bootstrap.cc:492] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:01.364352 775 tablet_copy_client.cc:523] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.365471 783 tablet_bootstrap.cc:654] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.366189 775 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:01.367223 648 tablet_copy_client.cc:523] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.367722 783 tablet_bootstrap.cc:492] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:01.368582 648 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:01.368777 783 ts_tablet_manager.cc:1367] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:01.369252 783 raft_consensus.cc:348] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.370726 783 raft_consensus.cc:374] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.370787 783 raft_consensus.cc:729] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.370903 783 consensus_queue.cc:260] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.373906 775 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.370980 31952 consensus_queue.cc:1035] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.374583 783 ts_tablet_manager.cc:1398] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:01.375644 775 tablet_bootstrap.cc:492] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:57:01.375823 775 ts_tablet_manager.cc:1367] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20240501 19:57:01.376276 775 raft_consensus.cc:348] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.376387 775 raft_consensus.cc:729] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.377279 775 consensus_queue.cc:260] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.377664 775 ts_tablet_manager.cc:1398] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.378392 27183 tablet_copy_service.cc:342] P 29d1a658e2b946e0b412e5fef82db9d9: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-92906e09f3474d7987082e0d33304886 received from {username='slave'} at 127.17.65.3:50333
I20240501 19:57:01.378490 27183 tablet_copy_service.cc:433] P 29d1a658e2b946e0b412e5fef82db9d9: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-92906e09f3474d7987082e0d33304886 on tablet 92906e09f3474d7987082e0d33304886 with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:01.382404 648 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.383332 32705 consensus_queue.cc:1035] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.385311 648 tablet_bootstrap.cc:492] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:01.385519 648 ts_tablet_manager.cc:1367] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240501 19:57:01.386032 648 raft_consensus.cc:348] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:01.386144 648 raft_consensus.cc:729] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.386296 648 consensus_queue.cc:260] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:01.387375 648 ts_tablet_manager.cc:1398] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.388072 27431 tablet_copy_service.cc:342] P 4a6e54e86a9949e5970b42e8e32e7561: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-2fed0e67f44543d09dbeda7100c7e0d6 received from {username='slave'} at 127.17.65.1:56347
I20240501 19:57:01.388168 27431 tablet_copy_service.cc:433] P 4a6e54e86a9949e5970b42e8e32e7561: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-2fed0e67f44543d09dbeda7100c7e0d6 on tablet 2fed0e67f44543d09dbeda7100c7e0d6 with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:01.392756 27163 raft_consensus.cc:1203] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:01.401546 444 raft_consensus.cc:482] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.401690 444 raft_consensus.cc:504] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.402094 444 leader_election.cc:288] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:01.402549 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbecd795f2f4471ab387b528a8b1aea3" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:01.402683 27032 raft_consensus.cc:2454] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:01.403134 24791 leader_election.cc:302] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:01.403335 444 raft_consensus.cc:2781] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.403398 444 raft_consensus.cc:482] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.403447 444 raft_consensus.cc:3037] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.404654 444 raft_consensus.cc:504] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.404966 444 leader_election.cc:288] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:01.405270 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbecd795f2f4471ab387b528a8b1aea3" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:01.405377 27032 raft_consensus.cc:3037] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.405822 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbecd795f2f4471ab387b528a8b1aea3" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:01.405941 27276 raft_consensus.cc:2454] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:01.406148 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbecd795f2f4471ab387b528a8b1aea3" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:01.406258 27278 raft_consensus.cc:3037] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.406505 27032 raft_consensus.cc:2454] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:01.406805 24791 leader_election.cc:302] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:01.407084 444 raft_consensus.cc:2781] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.407346 444 raft_consensus.cc:686] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:01.407742 27278 raft_consensus.cc:2454] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:01.407691 444 consensus_queue.cc:237] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:01.415133 24450 catalog_manager.cc:5484] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:01.423978 648 ts_tablet_manager.cc:902] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:01.424414 648 tablet_copy_client.cc:308] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:57:01.424950 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet 8eb1e61bb542415caac5978943e6b3eb from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:60025)
I20240501 19:57:01.425062 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet 8eb1e61bb542415caac5978943e6b3eb from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:60025: session id = bbf7735f82ca4f65a5170ac7909faa47-8eb1e61bb542415caac5978943e6b3eb
I20240501 19:57:01.425999 27052 tablet_copy_source_session.cc:205] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:01.426891 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb1e61bb542415caac5978943e6b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.429244 648 tablet_copy_client.cc:783] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:01.429435 648 tablet_copy_client.cc:647] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:01.430891 648 tablet_copy_client.cc:523] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:01.432292 648 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:01.436806 648 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:01.437050 648 tablet_bootstrap.cc:492] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:01.437208 648 ts_tablet_manager.cc:1367] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:57:01.437672 648 raft_consensus.cc:348] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:01.437790 648 raft_consensus.cc:729] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.437920 648 consensus_queue.cc:260] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:01.438196 648 ts_tablet_manager.cc:1398] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:01.439066 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-8eb1e61bb542415caac5978943e6b3eb received from {username='slave'} at 127.17.65.1:60025
I20240501 19:57:01.439142 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-8eb1e61bb542415caac5978943e6b3eb on tablet 8eb1e61bb542415caac5978943e6b3eb with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:01.499374 24594 raft_consensus.cc:1203] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:01.511152 24447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:01.516443 27424 raft_consensus.cc:1203] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:01.521459 27011 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:01.524423 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:01.526475 27143 tablet_service.cc:1467] Processing CreateTablet for tablet a73e21b4daff45a3990876f060497386 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.526821 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e21b4daff45a3990876f060497386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.527951 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 733a59323d4d434c89874d1dab8c9b06 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.528234 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733a59323d4d434c89874d1dab8c9b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.528208 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 733a59323d4d434c89874d1dab8c9b06 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.528476 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733a59323d4d434c89874d1dab8c9b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.530742 27267 tablet_service.cc:1467] Processing CreateTablet for tablet a54e2e2943ec41d785b5187f61cfbebf (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.530963 783 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:01.530982 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54e2e2943ec41d785b5187f61cfbebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.532128 783 tablet_bootstrap.cc:654] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.527951 24966 tablet_service.cc:1467] Processing CreateTablet for tablet a54e2e2943ec41d785b5187f61cfbebf (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.532900 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54e2e2943ec41d785b5187f61cfbebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.536996 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:01.538225 24705 tablet_service.cc:1467] Processing CreateTablet for tablet a73e21b4daff45a3990876f060497386 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.538225 27011 tablet_service.cc:1467] Processing CreateTablet for tablet a73e21b4daff45a3990876f060497386 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.538472 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e21b4daff45a3990876f060497386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.538471 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e21b4daff45a3990876f060497386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.539276 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 733a59323d4d434c89874d1dab8c9b06 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.539531 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733a59323d4d434c89874d1dab8c9b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.540804 24707 tablet_service.cc:1467] Processing CreateTablet for tablet a54e2e2943ec41d785b5187f61cfbebf (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.541025 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54e2e2943ec41d785b5187f61cfbebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.543062 776 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:01.544044 776 tablet_bootstrap.cc:654] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.545260 24447 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:01.547075 27267 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:01.547425 866 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:01.551403 24966 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:01.554075 27278 raft_consensus.cc:1261] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:01.555898 783 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:01.555989 783 ts_tablet_manager.cc:1367] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 19:57:01.556461 783 raft_consensus.cc:348] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.556569 783 raft_consensus.cc:374] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.556602 783 raft_consensus.cc:729] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.556763 783 consensus_queue.cc:260] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.557106 783 ts_tablet_manager.cc:1398] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.557272 783 tablet_bootstrap.cc:492] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:01.558178 783 tablet_bootstrap.cc:654] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.558461 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:01.559003 444 consensus_queue.cc:1035] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.559720 866 tablet_bootstrap.cc:654] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.564558 24837 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:01.564563 27032 raft_consensus.cc:1261] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:01.566146 444 consensus_queue.cc:1035] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.570977 24450 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet cbecd795f2f4471ab387b528a8b1aea3 (table rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029]). Expected version 1 got 0
I20240501 19:57:01.578980 24450 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet cbecd795f2f4471ab387b528a8b1aea3 (table rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029]). Expected version 1 got 0
I20240501 19:57:01.592893 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:01.599210 875 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:01.599753 875 tablet.cc:1620] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:01.602864 866 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:01.602964 866 ts_tablet_manager.cc:1367] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.000s
I20240501 19:57:01.603693 866 raft_consensus.cc:348] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.603847 866 raft_consensus.cc:374] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.603899 866 raft_consensus.cc:729] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.604044 866 consensus_queue.cc:260] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.604526 866 ts_tablet_manager.cc:1398] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.604832 866 tablet_bootstrap.cc:492] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:01.605973 866 tablet_bootstrap.cc:654] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.609081 24451 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet cbecd795f2f4471ab387b528a8b1aea3 (table rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029]). Expected version 1 got 0
I20240501 19:57:01.609542 776 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:01.609638 776 ts_tablet_manager.cc:1367] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.000s
I20240501 19:57:01.610092 776 raft_consensus.cc:348] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.610205 776 raft_consensus.cc:374] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.610239 776 raft_consensus.cc:729] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.610347 776 consensus_queue.cc:260] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.610615 776 ts_tablet_manager.cc:1398] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.610844 864 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:01.612287 864 tablet_bootstrap.cc:654] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.614575 868 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:01.611485 24450 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet cbecd795f2f4471ab387b528a8b1aea3 (table rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029]). Expected version 1 got 0
I20240501 19:57:01.615579 868 tablet_bootstrap.cc:654] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.622505 877 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:01.622866 877 tablet.cc:1620] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:01.626200 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029] alter complete (version 1)
I20240501 19:57:01.627621 864 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:01.629732 874 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:01.630005 783 tablet_bootstrap.cc:492] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:01.630048 864 ts_tablet_manager.cc:1367] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.000s
I20240501 19:57:01.630093 783 ts_tablet_manager.cc:1367] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.000s
I20240501 19:57:01.630539 864 raft_consensus.cc:348] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.630551 783 raft_consensus.cc:348] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:01.630652 864 raft_consensus.cc:374] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.630652 783 raft_consensus.cc:374] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.630686 864 raft_consensus.cc:729] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.630686 783 raft_consensus.cc:729] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.630810 783 consensus_queue.cc:260] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:01.631593 874 tablet.cc:1620] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:01.632225 783 ts_tablet_manager.cc:1398] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.632380 868 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:01.632473 868 ts_tablet_manager.cc:1367] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240501 19:57:01.632922 868 raft_consensus.cc:348] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.633031 868 raft_consensus.cc:374] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.633067 868 raft_consensus.cc:729] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.635082 864 consensus_queue.cc:260] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.638188 864 ts_tablet_manager.cc:1398] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:01.638394 868 consensus_queue.cc:260] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.639243 864 tablet_bootstrap.cc:492] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:01.640139 868 ts_tablet_manager.cc:1398] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240501 19:57:01.641134 864 tablet_bootstrap.cc:654] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.647745 864 tablet_bootstrap.cc:492] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:01.647879 866 tablet_bootstrap.cc:492] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:01.647893 864 ts_tablet_manager.cc:1367] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 19:57:01.647969 866 ts_tablet_manager.cc:1367] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.000s
I20240501 19:57:01.648309 864 raft_consensus.cc:348] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:01.648404 864 raft_consensus.cc:374] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.648363 866 raft_consensus.cc:348] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:01.648433 864 raft_consensus.cc:729] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.648466 866 raft_consensus.cc:374] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.648495 866 raft_consensus.cc:729] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.648608 866 consensus_queue.cc:260] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:01.648546 864 consensus_queue.cc:260] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:01.648866 866 ts_tablet_manager.cc:1398] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.649010 866 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:01.649080 864 ts_tablet_manager.cc:1398] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.649937 866 tablet_bootstrap.cc:654] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.658795 866 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:01.658891 866 ts_tablet_manager.cc:1367] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:57:01.659353 866 raft_consensus.cc:348] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.659454 866 raft_consensus.cc:374] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.659482 866 raft_consensus.cc:729] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.659592 866 consensus_queue.cc:260] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.659843 866 ts_tablet_manager.cc:1398] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:01.717470 24986 raft_consensus.cc:1203] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:01.731658 27163 raft_consensus.cc:1203] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:01.731890 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:01.735303 27032 raft_consensus.cc:1203] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:57:01.741473 874 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:01.741825 874 tablet.cc:1620] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:01.745857 875 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:01.746171 875 tablet.cc:1620] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:01.747017 877 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:01.747339 877 tablet.cc:1620] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:01.748579 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=37d6a11cb14e4a869cbd696d1ee84029] alter complete (version 2)
I20240501 19:57:01.749855 30967 raft_consensus.cc:482] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.749941 30967 raft_consensus.cc:504] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.750363 30967 leader_election.cc:288] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:01.750865 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e21b4daff45a3990876f060497386" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:01.751003 24727 raft_consensus.cc:2454] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:01.751349 26965 leader_election.cc:302] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:01.752913 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e21b4daff45a3990876f060497386" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:01.753038 27163 raft_consensus.cc:2454] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:01.754374 30967 raft_consensus.cc:2781] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.754447 30967 raft_consensus.cc:482] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.754490 30967 raft_consensus.cc:3037] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.755586 30967 raft_consensus.cc:504] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.755934 30967 leader_election.cc:288] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:01.757375 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e21b4daff45a3990876f060497386" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:01.757496 24727 raft_consensus.cc:3037] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.758564 24727 raft_consensus.cc:2454] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:01.759167 26965 leader_election.cc:302] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:01.760003 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e21b4daff45a3990876f060497386" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:01.760114 27163 raft_consensus.cc:3037] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.761111 30967 raft_consensus.cc:2781] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.761118 27163 raft_consensus.cc:2454] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:01.761438 30967 raft_consensus.cc:686] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:01.761569 30967 consensus_queue.cc:237] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.767985 24450 catalog_manager.cc:5484] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:01.777525 462 raft_consensus.cc:482] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.777634 462 raft_consensus.cc:504] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.778061 462 leader_election.cc:288] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:01.778381 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733a59323d4d434c89874d1dab8c9b06" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:01.778512 27278 raft_consensus.cc:2454] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:01.778544 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733a59323d4d434c89874d1dab8c9b06" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:01.778676 24727 raft_consensus.cc:2454] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:01.778879 24924 leader_election.cc:302] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:01.780117 462 raft_consensus.cc:2781] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.780181 462 raft_consensus.cc:482] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.780225 462 raft_consensus.cc:3037] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.781335 462 raft_consensus.cc:504] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.781661 462 leader_election.cc:288] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:01.782989 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733a59323d4d434c89874d1dab8c9b06" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:01.783136 27278 raft_consensus.cc:3037] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.784054 27278 raft_consensus.cc:2454] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:01.784395 24924 leader_election.cc:302] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:01.785007 462 raft_consensus.cc:2781] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.785104 462 raft_consensus.cc:686] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:01.785076 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733a59323d4d434c89874d1dab8c9b06" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:01.785184 24727 raft_consensus.cc:3037] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.785220 462 consensus_queue.cc:237] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:01.786274 24727 raft_consensus.cc:2454] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:01.787799 24449 catalog_manager.cc:5484] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:01.794498 24857 raft_consensus.cc:1203] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:01.853464 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:01.858023 27162 raft_consensus.cc:1203] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:01.859431 874 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:01.859988 877 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:01.863052 875 tablet.cc:1722] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:01.866585 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=37d6a11cb14e4a869cbd696d1ee84029] alter complete (version 3)
I20240501 19:57:01.909451 24594 raft_consensus.cc:1203] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:01.968575 24449 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:01.968752 24449 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:01.970801 24449 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet cbecd795f2f4471ab387b528a8b1aea3
I20240501 19:57:01.971374 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet cbecd795f2f4471ab387b528a8b1aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:01 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:01.971439 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet cbecd795f2f4471ab387b528a8b1aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:01 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:01.971503 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet cbecd795f2f4471ab387b528a8b1aea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:01 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:01.971787 927 tablet_replica.cc:330] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:01.971840 928 tablet_replica.cc:330] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:01.971881 926 tablet_replica.cc:330] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:01.972024 927 raft_consensus.cc:2229] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:01.972047 928 raft_consensus.cc:2229] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:01.972041 926 raft_consensus.cc:2229] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:01.972230 926 raft_consensus.cc:2258] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:01.972239 928 raft_consensus.cc:2258] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:01.972255 927 raft_consensus.cc:2258] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:01.973712 927 ts_tablet_manager.cc:1875] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:01.973742 928 ts_tablet_manager.cc:1875] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:01.973776 926 ts_tablet_manager.cc:1875] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:01.976115 928 ts_tablet_manager.cc:1888] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:01.976222 928 log.cc:1196] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/cbecd795f2f4471ab387b528a8b1aea3
I20240501 19:57:01.976255 927 ts_tablet_manager.cc:1888] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:01.976352 927 log.cc:1196] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/cbecd795f2f4471ab387b528a8b1aea3
I20240501 19:57:01.976536 928 ts_tablet_manager.cc:1909] T cbecd795f2f4471ab387b528a8b1aea3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:57:01.976672 927 ts_tablet_manager.cc:1909] T cbecd795f2f4471ab387b528a8b1aea3 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:01.977023 926 ts_tablet_manager.cc:1888] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:01.977133 926 log.cc:1196] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/cbecd795f2f4471ab387b528a8b1aea3
I20240501 19:57:01.977216 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet cbecd795f2f4471ab387b528a8b1aea3 (table rebalancer_dynamic_table_new_name [id=37d6a11cb14e4a869cbd696d1ee84029]) successfully deleted
I20240501 19:57:01.977398 926 ts_tablet_manager.cc:1909] T cbecd795f2f4471ab387b528a8b1aea3 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:57:01.977499 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet cbecd795f2f4471ab387b528a8b1aea3 (table rebalancer_dynamic_table_new_name [id=37d6a11cb14e4a869cbd696d1ee84029]) successfully deleted
I20240501 19:57:01.978031 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet cbecd795f2f4471ab387b528a8b1aea3 (table rebalancer_dynamic_table_new_name [id=37d6a11cb14e4a869cbd696d1ee84029]) successfully deleted
I20240501 19:57:01.985282 24594 raft_consensus.cc:1203] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:02.039388 32475 raft_consensus.cc:482] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.039527 32475 raft_consensus.cc:504] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:02.039969 32475 leader_election.cc:288] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:02.040344 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54e2e2943ec41d785b5187f61cfbebf" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:02.040325 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54e2e2943ec41d785b5187f61cfbebf" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:02.040474 27278 raft_consensus.cc:2454] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:02.040488 24986 raft_consensus.cc:2454] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:02.040750 24662 leader_election.cc:302] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:02.040894 32475 raft_consensus.cc:2781] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:02.040956 32475 raft_consensus.cc:482] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:02.040995 32475 raft_consensus.cc:3037] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.042137 32475 raft_consensus.cc:504] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:02.042492 32475 leader_election.cc:288] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:02.042769 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54e2e2943ec41d785b5187f61cfbebf" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:02.042814 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54e2e2943ec41d785b5187f61cfbebf" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:02.042894 27278 raft_consensus.cc:3037] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.042927 24986 raft_consensus.cc:3037] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.043761 24986 raft_consensus.cc:2454] T a54e2e2943ec41d785b5187f61cfbebf P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:02.043999 27278 raft_consensus.cc:2454] T a54e2e2943ec41d785b5187f61cfbebf P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:02.044011 24661 leader_election.cc:302] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:02.044159 32475 raft_consensus.cc:2781] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:02.044237 32475 raft_consensus.cc:686] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:02.044574 32475 consensus_queue.cc:237] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:02.046432 24449 catalog_manager.cc:5484] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:02.069958 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:02.072337 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:02.073874 24449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:02.080466 27011 tablet_service.cc:1467] Processing CreateTablet for tablet b09b0d9246834e91a64bea1c634ef01e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ddca474c5b254d9d92f152b6ac967591]), partition=
I20240501 19:57:02.080726 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09b0d9246834e91a64bea1c634ef01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.082823 776 tablet_bootstrap.cc:492] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:02.083001 27267 tablet_service.cc:1467] Processing CreateTablet for tablet b09b0d9246834e91a64bea1c634ef01e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ddca474c5b254d9d92f152b6ac967591]), partition=
I20240501 19:57:02.083287 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09b0d9246834e91a64bea1c634ef01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.083704 776 tablet_bootstrap.cc:654] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.085036 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:02.085237 783 tablet_bootstrap.cc:492] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:02.085368 27405 tablet_service.cc:1467] Processing CreateTablet for tablet b09b0d9246834e91a64bea1c634ef01e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ddca474c5b254d9d92f152b6ac967591]), partition=
I20240501 19:57:02.085597 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09b0d9246834e91a64bea1c634ef01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.086228 783 tablet_bootstrap.cc:654] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.086547 24966 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:02.088022 950 tablet_bootstrap.cc:492] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:02.088809 27405 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:02.089068 776 tablet_bootstrap.cc:492] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:02.089144 776 ts_tablet_manager.cc:1367] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:02.089537 776 raft_consensus.cc:348] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.089645 776 raft_consensus.cc:374] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.089671 776 raft_consensus.cc:729] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.089772 776 consensus_queue.cc:260] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.090010 776 ts_tablet_manager.cc:1398] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:02.090284 950 tablet_bootstrap.cc:654] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.091535 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:02.092453 783 tablet_bootstrap.cc:492] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:02.092540 783 ts_tablet_manager.cc:1367] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:02.092988 783 raft_consensus.cc:348] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.093086 783 raft_consensus.cc:374] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.093114 783 raft_consensus.cc:729] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.093217 783 consensus_queue.cc:260] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.093473 783 ts_tablet_manager.cc:1398] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:02.101996 27267 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:02.104310 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:02.135596 950 tablet_bootstrap.cc:492] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:02.135727 950 ts_tablet_manager.cc:1367] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.000s
I20240501 19:57:02.136205 950 raft_consensus.cc:348] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.136333 950 raft_consensus.cc:374] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.136385 950 raft_consensus.cc:729] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.136514 950 consensus_queue.cc:260] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.136826 950 ts_tablet_manager.cc:1398] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:02.145454 929 raft_consensus.cc:482] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.145565 929 raft_consensus.cc:504] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.145946 929 leader_election.cc:288] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:02.146514 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09b0d9246834e91a64bea1c634ef01e" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:02.146653 27032 raft_consensus.cc:2454] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:02.146926 27359 leader_election.cc:302] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:02.146935 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09b0d9246834e91a64bea1c634ef01e" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:02.147090 27278 raft_consensus.cc:2454] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:02.147142 929 raft_consensus.cc:2781] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:02.147193 929 raft_consensus.cc:482] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:02.147235 929 raft_consensus.cc:3037] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.148346 929 raft_consensus.cc:504] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.148675 929 leader_election.cc:288] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:02.148953 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09b0d9246834e91a64bea1c634ef01e" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:02.149053 27032 raft_consensus.cc:3037] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.149166 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09b0d9246834e91a64bea1c634ef01e" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:02.149295 27278 raft_consensus.cc:3037] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.150043 27032 raft_consensus.cc:2454] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:02.150228 27278 raft_consensus.cc:2454] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:02.150768 27359 leader_election.cc:302] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:02.150916 929 raft_consensus.cc:2781] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:02.151011 929 raft_consensus.cc:686] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:57:02.151453 929 consensus_queue.cc:237] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.157372 24449 catalog_manager.cc:5484] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:02.176052 30967 consensus_queue.cc:1035] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.184422 24856 consensus_queue.cc:237] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.185669 27032 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:02.185715 27162 raft_consensus.cc:1261] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:02.186060 925 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:02.186175 444 consensus_queue.cc:1035] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:02.187918 815 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:02.188251 27032 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:02.189942 27162 raft_consensus.cc:2932] T 6d2603a318c34256a9fa28979ae41fec P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:02.191623 24450 catalog_manager.cc:5484] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 6 to 7, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:02.193053 462 consensus_queue.cc:1035] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.196028 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2603a318c34256a9fa28979ae41fec with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:02.196431 974 tablet_replica.cc:330] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:02.196372 587 consensus_queue.cc:1035] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.196939 974 raft_consensus.cc:2229] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:02.197142 974 raft_consensus.cc:2258] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:02.200073 974 ts_tablet_manager.cc:1875] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:02.202576 462 consensus_queue.cc:1035] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.203619 974 ts_tablet_manager.cc:1888] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:02.203724 974 log.cc:1196] T 6d2603a318c34256a9fa28979ae41fec P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d2603a318c34256a9fa28979ae41fec
I20240501 19:57:02.204561 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 6d2603a318c34256a9fa28979ae41fec (table rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]) successfully deleted
I20240501 19:57:02.263425 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:02.274037 27278 raft_consensus.cc:1261] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:02.274251 27032 raft_consensus.cc:1261] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:02.274454 419 consensus_queue.cc:1035] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.274657 929 consensus_queue.cc:1035] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.277451 24450 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet b09b0d9246834e91a64bea1c634ef01e (table rebalancer_dynamic_table [id=ddca474c5b254d9d92f152b6ac967591]). Expected version 1 got 0
I20240501 19:57:02.277873 24453 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet b09b0d9246834e91a64bea1c634ef01e (table rebalancer_dynamic_table [id=ddca474c5b254d9d92f152b6ac967591]). Expected version 1 got 0
I20240501 19:57:02.280154 875 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:02.280464 875 tablet.cc:1620] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:02.280515 877 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:02.280776 877 tablet.cc:1620] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:02.282428 24453 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet b09b0d9246834e91a64bea1c634ef01e (table rebalancer_dynamic_table [id=ddca474c5b254d9d92f152b6ac967591]). Expected version 1 got 0
I20240501 19:57:02.285184 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=ddca474c5b254d9d92f152b6ac967591] alter complete (version 1)
I20240501 19:57:02.293215 1026 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:02.294483 1026 tablet.cc:1620] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:02.317167 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:02.320367 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:02.321755 27267 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:02.325748 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:02.333429 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:02.345331 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:02.345650 27405 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:02.362172 24966 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:02.389839 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:02.395099 1026 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:02.395470 1026 tablet.cc:1620] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:02.395571 875 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:02.395869 875 tablet.cc:1620] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:02.396113 877 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:02.396452 877 tablet.cc:1620] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:02.402724 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=ddca474c5b254d9d92f152b6ac967591] alter complete (version 2)
I20240501 19:57:02.434931 24856 consensus_queue.cc:237] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:02.436280 27163 raft_consensus.cc:1261] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:02.436271 24986 raft_consensus.cc:1261] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:02.436673 444 consensus_queue.cc:1035] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:02.436864 815 consensus_queue.cc:1035] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:02.438535 925 raft_consensus.cc:2932] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:02.438997 27163 raft_consensus.cc:2932] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:02.441756 24447 catalog_manager.cc:5484] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:02.443603 24986 raft_consensus.cc:2932] T 2c52abe3cfff44f88f41e1e885e5f2c9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:02.445297 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c52abe3cfff44f88f41e1e885e5f2c9 with delete_type TABLET_DATA_TOMBSTONED (TS 53e34bb09d8e41129a5683079ff63610 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:02.445576 926 tablet_replica.cc:330] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:02.445745 926 raft_consensus.cc:2229] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:02.445897 926 raft_consensus.cc:2258] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:02.446336 926 ts_tablet_manager.cc:1875] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:02.448137 926 ts_tablet_manager.cc:1888] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:02.448194 926 log.cc:1196] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/2c52abe3cfff44f88f41e1e885e5f2c9
I20240501 19:57:02.448698 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 2c52abe3cfff44f88f41e1e885e5f2c9 (table rebalancer_extra_table_14 [id=1372309a69704991b2ebb0365b3700a6]) successfully deleted
I20240501 19:57:02.506791 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:02.514405 1026 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:02.515504 875 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:02.515566 877 tablet.cc:1722] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:02.520097 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=ddca474c5b254d9d92f152b6ac967591] alter complete (version 3)
I20240501 19:57:02.547108 32253 consensus_queue.cc:1035] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.552529 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:02.552945 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:02.555855 27405 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:02.566303 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:02.570724 32475 consensus_queue.cc:1035] T a54e2e2943ec41d785b5187f61cfbebf P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.579738 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:02.586217 27267 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:02.591382 24966 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:02.611114 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:02.621798 24450 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:02.621965 24450 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:02.624202 24450 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet b09b0d9246834e91a64bea1c634ef01e
I20240501 19:57:02.625500 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet b09b0d9246834e91a64bea1c634ef01e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:02 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:02.625641 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet b09b0d9246834e91a64bea1c634ef01e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:02 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:02.625830 926 tablet_replica.cc:330] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:02.625989 926 raft_consensus.cc:2229] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:02.626128 926 raft_consensus.cc:2258] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:02.626911 926 ts_tablet_manager.cc:1875] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:02.629120 926 ts_tablet_manager.cc:1888] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:02.629177 926 log.cc:1196] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/b09b0d9246834e91a64bea1c634ef01e
I20240501 19:57:02.630529 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet b09b0d9246834e91a64bea1c634ef01e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:02 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:02.631178 926 ts_tablet_manager.cc:1909] T b09b0d9246834e91a64bea1c634ef01e P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:57:02.632825 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet b09b0d9246834e91a64bea1c634ef01e (table rebalancer_dynamic_table_new_name [id=ddca474c5b254d9d92f152b6ac967591]) successfully deleted
I20240501 19:57:02.634313 1112 tablet_replica.cc:330] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:02.634487 1112 raft_consensus.cc:2229] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:02.634608 1112 raft_consensus.cc:2258] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:02.635202 1112 ts_tablet_manager.cc:1875] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:02.636386 1111 tablet_replica.cc:330] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:02.636562 1111 raft_consensus.cc:2229] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:02.636797 1111 raft_consensus.cc:2258] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:02.637630 1112 ts_tablet_manager.cc:1888] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:02.637746 1112 log.cc:1196] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/b09b0d9246834e91a64bea1c634ef01e
I20240501 19:57:02.638036 1112 ts_tablet_manager.cc:1909] T b09b0d9246834e91a64bea1c634ef01e P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:02.638686 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet b09b0d9246834e91a64bea1c634ef01e (table rebalancer_dynamic_table_new_name [id=ddca474c5b254d9d92f152b6ac967591]) successfully deleted
I20240501 19:57:02.639855 1111 ts_tablet_manager.cc:1875] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:02.642076 1111 ts_tablet_manager.cc:1888] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:02.642175 1111 log.cc:1196] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/b09b0d9246834e91a64bea1c634ef01e
I20240501 19:57:02.642498 1111 ts_tablet_manager.cc:1909] T b09b0d9246834e91a64bea1c634ef01e P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:57:02.643198 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet b09b0d9246834e91a64bea1c634ef01e (table rebalancer_dynamic_table_new_name [id=ddca474c5b254d9d92f152b6ac967591]) successfully deleted
I20240501 19:57:02.655565 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:02.663401 24966 tablet_service.cc:1467] Processing CreateTablet for tablet c35b65e8bb074948b55d675b806b41d8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.663731 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35b65e8bb074948b55d675b806b41d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.663754 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 5d80b9d644fb4160950462cdcc895052 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.664011 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d80b9d644fb4160950462cdcc895052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.664276 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 79f6ba04101645aa9b5e894b9db07bf5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.664276 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 5d80b9d644fb4160950462cdcc895052 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.664557 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d80b9d644fb4160950462cdcc895052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.664582 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f6ba04101645aa9b5e894b9db07bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.666414 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 79f6ba04101645aa9b5e894b9db07bf5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.666739 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f6ba04101645aa9b5e894b9db07bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.666932 27143 tablet_service.cc:1467] Processing CreateTablet for tablet c35b65e8bb074948b55d675b806b41d8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.667165 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35b65e8bb074948b55d675b806b41d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.669200 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 5d80b9d644fb4160950462cdcc895052 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.669469 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d80b9d644fb4160950462cdcc895052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.663754 24574 tablet_service.cc:1467] Processing CreateTablet for tablet c35b65e8bb074948b55d675b806b41d8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.670276 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35b65e8bb074948b55d675b806b41d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.671785 1128 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:02.672884 1123 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:02.673333 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 79f6ba04101645aa9b5e894b9db07bf5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:02.673604 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f6ba04101645aa9b5e894b9db07bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.673782 1119 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:02.674984 1120 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:02.675323 1118 tablet_bootstrap.cc:492] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:02.676576 1123 tablet_bootstrap.cc:654] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.677819 1118 tablet_bootstrap.cc:654] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.677981 1128 tablet_bootstrap.cc:654] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.678694 1121 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:02.679198 1128 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:02.679286 1128 ts_tablet_manager.cc:1367] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:57:02.679425 1119 tablet_bootstrap.cc:654] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.679795 1128 raft_consensus.cc:348] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.679899 1128 raft_consensus.cc:374] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.679940 1128 raft_consensus.cc:729] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.680084 1120 tablet_bootstrap.cc:654] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.680075 1128 consensus_queue.cc:260] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.680361 1128 ts_tablet_manager.cc:1398] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:02.681808 1121 tablet_bootstrap.cc:654] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.683398 1119 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:02.683817 1119 ts_tablet_manager.cc:1367] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:57:02.684425 1119 raft_consensus.cc:348] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.684943 1121 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:02.685037 1119 raft_consensus.cc:374] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.685124 1119 raft_consensus.cc:729] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.685286 1119 consensus_queue.cc:260] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.685772 1120 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:02.685904 1120 ts_tablet_manager.cc:1367] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:02.686563 1120 raft_consensus.cc:348] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.687173 1121 ts_tablet_manager.cc:1367] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 19:57:02.687646 1119 ts_tablet_manager.cc:1398] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:02.687713 1121 raft_consensus.cc:348] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.687830 1119 tablet_bootstrap.cc:492] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:02.687836 1121 raft_consensus.cc:374] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.687929 1121 raft_consensus.cc:729] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.688066 1121 consensus_queue.cc:260] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.688329 1121 ts_tablet_manager.cc:1398] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:02.688800 1119 tablet_bootstrap.cc:654] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.689249 1118 tablet_bootstrap.cc:492] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:02.689391 1118 ts_tablet_manager.cc:1367] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:57:02.689581 1123 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:02.689832 1123 ts_tablet_manager.cc:1367] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:57:02.690531 1123 raft_consensus.cc:348] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.690701 1123 raft_consensus.cc:374] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.690666 1118 raft_consensus.cc:348] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.690769 1118 raft_consensus.cc:374] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.690807 1123 raft_consensus.cc:729] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.690858 1118 raft_consensus.cc:729] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.690990 1123 consensus_queue.cc:260] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.691078 1120 raft_consensus.cc:374] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.691131 1120 raft_consensus.cc:729] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.691243 1118 consensus_queue.cc:260] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.691504 1120 consensus_queue.cc:260] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.691891 1118 ts_tablet_manager.cc:1398] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:02.693022 1120 ts_tablet_manager.cc:1398] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:02.693295 1120 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:02.693634 1123 ts_tablet_manager.cc:1398] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:02.694355 1120 tablet_bootstrap.cc:654] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.695188 1123 tablet_bootstrap.cc:492] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:02.696285 1119 tablet_bootstrap.cc:492] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:02.696377 1119 ts_tablet_manager.cc:1367] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:02.696763 1119 raft_consensus.cc:348] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.696744 24986 consensus_queue.cc:237] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.696848 1123 tablet_bootstrap.cc:654] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.696852 1119 raft_consensus.cc:374] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.696889 1119 raft_consensus.cc:729] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.697047 1119 consensus_queue.cc:260] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.697330 1119 ts_tablet_manager.cc:1398] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:02.697997 1123 tablet_bootstrap.cc:492] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:02.698146 24594 raft_consensus.cc:1261] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:02.698647 1120 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:02.698803 1120 ts_tablet_manager.cc:1367] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:02.698881 731 consensus_queue.cc:1035] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:02.698226 27162 raft_consensus.cc:1261] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:02.700233 1123 ts_tablet_manager.cc:1367] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:02.700428 731 consensus_queue.cc:1035] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:02.700917 1120 raft_consensus.cc:348] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.701069 1120 raft_consensus.cc:374] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.701146 1120 raft_consensus.cc:729] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.701315 1120 consensus_queue.cc:260] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.701751 1120 ts_tablet_manager.cc:1398] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:02.700637 1123 raft_consensus.cc:348] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.701867 1123 raft_consensus.cc:374] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.701913 1123 raft_consensus.cc:729] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.701824 731 raft_consensus.cc:2932] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:02.702046 1123 consensus_queue.cc:260] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.702304 1123 ts_tablet_manager.cc:1398] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 19:57:02.704133 27162 raft_consensus.cc:2932] T 53981ad3756d4c54994f7aea973c5cde P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:02.705178 24446 catalog_manager.cc:5484] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:02.705298 24594 raft_consensus.cc:2932] T 53981ad3756d4c54994f7aea973c5cde P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:02.707687 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 53981ad3756d4c54994f7aea973c5cde with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:02.709301 1147 tablet_replica.cc:330] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:02.709517 1147 raft_consensus.cc:2229] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:02.709717 1147 raft_consensus.cc:2258] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:02.710215 1147 ts_tablet_manager.cc:1875] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:02.712271 1147 ts_tablet_manager.cc:1888] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:02.712335 1147 log.cc:1196] T 53981ad3756d4c54994f7aea973c5cde P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/53981ad3756d4c54994f7aea973c5cde
I20240501 19:57:02.712822 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 53981ad3756d4c54994f7aea973c5cde (table rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]) successfully deleted
I20240501 19:57:02.716589 1146 raft_consensus.cc:482] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.716719 1146 raft_consensus.cc:504] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.717139 1146 leader_election.cc:288] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:02.717603 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35b65e8bb074948b55d675b806b41d8" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:02.717744 24594 raft_consensus.cc:2454] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:02.718070 24922 leader_election.cc:302] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:02.718438 731 raft_consensus.cc:2781] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:02.718489 731 raft_consensus.cc:482] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:02.718528 731 raft_consensus.cc:3037] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.718508 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35b65e8bb074948b55d675b806b41d8" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:02.718626 27162 raft_consensus.cc:2454] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:02.719573 731 raft_consensus.cc:504] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.719940 731 leader_election.cc:288] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:02.720245 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35b65e8bb074948b55d675b806b41d8" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:02.720358 27162 raft_consensus.cc:3037] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.720337 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35b65e8bb074948b55d675b806b41d8" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:02.720433 24594 raft_consensus.cc:3037] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.721148 27162 raft_consensus.cc:2454] T c35b65e8bb074948b55d675b806b41d8 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:02.721253 24594 raft_consensus.cc:2454] T c35b65e8bb074948b55d675b806b41d8 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:02.721431 24924 leader_election.cc:302] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:02.721588 731 raft_consensus.cc:2781] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:02.721662 731 raft_consensus.cc:686] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:02.721784 731 consensus_queue.cc:237] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:02.724344 24449 catalog_manager.cc:5484] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:02.726649 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:02.731611 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 05c2d3abee2e4610bed4ee58e9153df5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da7daf89e87b4602824738ea0b6ac070]), partition=
I20240501 19:57:02.731894 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c2d3abee2e4610bed4ee58e9153df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.731981 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 05c2d3abee2e4610bed4ee58e9153df5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da7daf89e87b4602824738ea0b6ac070]), partition=
I20240501 19:57:02.732213 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c2d3abee2e4610bed4ee58e9153df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.732872 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 05c2d3abee2e4610bed4ee58e9153df5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da7daf89e87b4602824738ea0b6ac070]), partition=
I20240501 19:57:02.733095 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c2d3abee2e4610bed4ee58e9153df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:02.733645 1121 tablet_bootstrap.cc:492] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:02.734288 1123 tablet_bootstrap.cc:492] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:02.734575 1121 tablet_bootstrap.cc:654] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.735236 1123 tablet_bootstrap.cc:654] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.735260 1161 tablet_bootstrap.cc:492] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:02.736547 1161 tablet_bootstrap.cc:654] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:02.736567 1123 tablet_bootstrap.cc:492] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:02.736640 1123 ts_tablet_manager.cc:1367] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:02.737042 1123 raft_consensus.cc:348] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.737146 1123 raft_consensus.cc:374] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.737190 1123 raft_consensus.cc:729] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.737316 1123 consensus_queue.cc:260] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.737577 1123 ts_tablet_manager.cc:1398] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:02.737602 1161 tablet_bootstrap.cc:492] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:02.737679 1161 ts_tablet_manager.cc:1367] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:02.738202 1161 raft_consensus.cc:348] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.738292 1161 raft_consensus.cc:374] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.738322 1161 raft_consensus.cc:729] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.738451 1161 consensus_queue.cc:260] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.738832 1161 ts_tablet_manager.cc:1398] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:02.744046 1121 tablet_bootstrap.cc:492] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:02.744140 1121 ts_tablet_manager.cc:1367] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240501 19:57:02.744545 1121 raft_consensus.cc:348] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.744658 1121 raft_consensus.cc:374] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:02.744700 1121 raft_consensus.cc:729] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.744819 1121 consensus_queue.cc:260] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.745085 1121 ts_tablet_manager.cc:1398] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:02.769861 561 raft_consensus.cc:482] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.769991 561 raft_consensus.cc:504] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.770499 561 leader_election.cc:288] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:02.770761 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c2d3abee2e4610bed4ee58e9153df5" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:02.770887 27276 raft_consensus.cc:2454] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:02.770906 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c2d3abee2e4610bed4ee58e9153df5" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:02.771067 27031 raft_consensus.cc:2454] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:02.771247 27098 leader_election.cc:302] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:02.771421 561 raft_consensus.cc:2781] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:02.771471 561 raft_consensus.cc:482] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:02.771512 561 raft_consensus.cc:3037] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.772292 1038 raft_consensus.cc:482] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.772387 1038 raft_consensus.cc:504] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.772508 561 raft_consensus.cc:504] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.772796 561 leader_election.cc:288] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:02.772889 1038 leader_election.cc:288] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:02.773236 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f6ba04101645aa9b5e894b9db07bf5" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:02.773345 27162 raft_consensus.cc:2454] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:02.773607 27229 leader_election.cc:302] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:02.773840 1038 raft_consensus.cc:2781] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:02.773825 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f6ba04101645aa9b5e894b9db07bf5" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:02.773912 1038 raft_consensus.cc:482] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:02.773962 1038 raft_consensus.cc:3037] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.773967 24856 raft_consensus.cc:2454] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:02.774541 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c2d3abee2e4610bed4ee58e9153df5" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:02.774629 27276 raft_consensus.cc:3037] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.775245 1038 raft_consensus.cc:504] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.775564 27276 raft_consensus.cc:2454] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:02.775684 1038 leader_election.cc:288] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:02.776608 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f6ba04101645aa9b5e894b9db07bf5" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:02.776717 27162 raft_consensus.cc:3037] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.777593 27162 raft_consensus.cc:2454] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:02.777944 27229 leader_election.cc:302] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:02.777966 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c2d3abee2e4610bed4ee58e9153df5" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:02.778079 27031 raft_consensus.cc:3037] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.778177 1038 raft_consensus.cc:2781] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:02.778250 1038 raft_consensus.cc:686] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:02.778383 1038 consensus_queue.cc:237] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:02.779013 27031 raft_consensus.cc:2454] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:02.779314 27096 leader_election.cc:302] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:57:02.780158 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f6ba04101645aa9b5e894b9db07bf5" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:02.780262 24856 raft_consensus.cc:3037] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.781173 24856 raft_consensus.cc:2454] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:02.781106 24450 catalog_manager.cc:5484] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:02.782294 561 raft_consensus.cc:2781] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:02.782385 561 raft_consensus.cc:686] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:02.782500 561 consensus_queue.cc:237] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:02.784824 24446 catalog_manager.cc:5484] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
W20240501 19:57:02.795091 24643 tablet.cc:2369] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:57:02.833925 25036 tablet.cc:2369] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:02.893899 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:02.897938 27031 raft_consensus.cc:1261] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:02.898025 27276 raft_consensus.cc:1261] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:02.898336 561 consensus_queue.cc:1035] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.898497 977 consensus_queue.cc:1035] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.901405 1201 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:02.901461 24451 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 05c2d3abee2e4610bed4ee58e9153df5 (table rebalancer_dynamic_table [id=da7daf89e87b4602824738ea0b6ac070]). Expected version 1 got 0
I20240501 19:57:02.901818 1201 tablet.cc:1620] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:02.901921 877 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:02.902051 875 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:02.902199 877 tablet.cc:1620] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:02.902246 1197 mvcc.cc:205] Tried to move back new op lower bound from 7022974660186542080 to 7022974659718258688. Current Snapshot: MvccSnapshot[applied={T|T < 7022974660186542080}]
I20240501 19:57:02.902297 875 tablet.cc:1620] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:02.907171 24445 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet 05c2d3abee2e4610bed4ee58e9153df5 (table rebalancer_dynamic_table [id=da7daf89e87b4602824738ea0b6ac070]). Expected version 1 got 0
I20240501 19:57:02.909518 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=da7daf89e87b4602824738ea0b6ac070] alter complete (version 1)
W20240501 19:57:02.913378 24905 tablet.cc:2369] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:02.955484 929 raft_consensus.cc:482] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.955690 929 raft_consensus.cc:504] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.956336 929 leader_election.cc:288] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:02.956710 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d80b9d644fb4160950462cdcc895052" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:02.956864 24857 raft_consensus.cc:2454] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:02.956877 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d80b9d644fb4160950462cdcc895052" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:02.957041 24594 raft_consensus.cc:2454] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:02.957369 27360 leader_election.cc:302] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:02.957662 929 raft_consensus.cc:2781] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:02.957726 929 raft_consensus.cc:482] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:02.957777 929 raft_consensus.cc:3037] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.958846 929 raft_consensus.cc:504] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.959200 929 leader_election.cc:288] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:02.959626 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d80b9d644fb4160950462cdcc895052" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:02.959745 24857 raft_consensus.cc:3037] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.959734 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d80b9d644fb4160950462cdcc895052" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:02.959848 24594 raft_consensus.cc:3037] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.960671 24594 raft_consensus.cc:2454] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:02.960753 24857 raft_consensus.cc:2454] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:02.961236 27360 leader_election.cc:302] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:02.961434 929 raft_consensus.cc:2781] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:02.961535 929 raft_consensus.cc:686] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:57:02.961683 929 consensus_queue.cc:237] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:02.963797 24451 catalog_manager.cc:5484] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:02.992759 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:02.995258 27267 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:02.995756 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:02.997321 24966 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:02.997696 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:03.003408 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:03.014842 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:03.015331 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:03.019167 27405 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:03.021307 1201 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:03.021629 1201 tablet.cc:1620] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:03.021646 875 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:03.021895 875 tablet.cc:1620] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:03.026836 877 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:03.027124 877 tablet.cc:1620] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:03.032241 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=da7daf89e87b4602824738ea0b6ac070] alter complete (version 2)
I20240501 19:57:03.089944 24857 consensus_queue.cc:237] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.091317 27424 raft_consensus.cc:1261] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:03.091923 24725 raft_consensus.cc:1261] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:03.092319 444 consensus_queue.cc:1035] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:03.092808 444 consensus_queue.cc:1035] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:03.094168 925 raft_consensus.cc:2932] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:03.094625 24725 raft_consensus.cc:2932] T cfb037c9bf1a4ed080745df687ac4a89 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:03.095016 27424 raft_consensus.cc:2932] T cfb037c9bf1a4ed080745df687ac4a89 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:03.098507 24451 catalog_manager.cc:5484] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 5, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:03.100957 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb037c9bf1a4ed080745df687ac4a89 with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:03.101333 1238 tablet_replica.cc:330] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:03.101536 1238 raft_consensus.cc:2229] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:03.101713 1238 raft_consensus.cc:2258] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:03.102290 1238 ts_tablet_manager.cc:1875] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:03.104975 1238 ts_tablet_manager.cc:1888] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:03.105048 1238 log.cc:1196] T cfb037c9bf1a4ed080745df687ac4a89 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/cfb037c9bf1a4ed080745df687ac4a89
I20240501 19:57:03.105597 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet cfb037c9bf1a4ed080745df687ac4a89 (table rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]) successfully deleted
I20240501 19:57:03.118785 1146 consensus_queue.cc:1035] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.123173 731 consensus_queue.cc:1035] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.134085 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:03.145156 1201 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:03.148694 875 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:03.149646 877 tablet.cc:1722] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:03.161618 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=da7daf89e87b4602824738ea0b6ac070] alter complete (version 3)
I20240501 19:57:03.224387 1038 consensus_queue.cc:1035] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.230605 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:03.231695 27405 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:03.231951 32705 consensus_queue.cc:1035] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.233322 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:03.236330 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:03.257097 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:03.261430 27267 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:03.266278 24966 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:03.268791 24574 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:03.275246 24450 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:03.275403 24450 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:03.278210 24450 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 05c2d3abee2e4610bed4ee58e9153df5
I20240501 19:57:03.278728 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet 05c2d3abee2e4610bed4ee58e9153df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:03 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:03.278901 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet 05c2d3abee2e4610bed4ee58e9153df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:03 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:03.278990 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 05c2d3abee2e4610bed4ee58e9153df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:03 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:03.286381 1295 tablet_replica.cc:330] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:03.286577 1295 raft_consensus.cc:2229] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:03.286712 1295 raft_consensus.cc:2258] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:03.286929 1294 tablet_replica.cc:330] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:03.287446 1294 raft_consensus.cc:2229] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:03.287622 1294 raft_consensus.cc:2258] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:03.283869 1296 tablet_replica.cc:330] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:03.290540 1296 raft_consensus.cc:2229] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:03.290796 1296 raft_consensus.cc:2258] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:03.292613 1296 ts_tablet_manager.cc:1875] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:03.293524 1294 ts_tablet_manager.cc:1875] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:03.293927 1295 ts_tablet_manager.cc:1875] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:03.295249 1296 ts_tablet_manager.cc:1888] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:03.295318 1296 log.cc:1196] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/05c2d3abee2e4610bed4ee58e9153df5
I20240501 19:57:03.295641 1296 ts_tablet_manager.cc:1909] T 05c2d3abee2e4610bed4ee58e9153df5 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:57:03.296721 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 05c2d3abee2e4610bed4ee58e9153df5 (table rebalancer_dynamic_table_new_name [id=da7daf89e87b4602824738ea0b6ac070]) successfully deleted
I20240501 19:57:03.299806 1294 ts_tablet_manager.cc:1888] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:03.299932 1294 log.cc:1196] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/05c2d3abee2e4610bed4ee58e9153df5
I20240501 19:57:03.300176 1294 ts_tablet_manager.cc:1909] T 05c2d3abee2e4610bed4ee58e9153df5 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:57:03.300155 1295 ts_tablet_manager.cc:1888] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:03.300233 1295 log.cc:1196] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/05c2d3abee2e4610bed4ee58e9153df5
I20240501 19:57:03.300559 1295 ts_tablet_manager.cc:1909] T 05c2d3abee2e4610bed4ee58e9153df5 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:03.300905 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 05c2d3abee2e4610bed4ee58e9153df5 (table rebalancer_dynamic_table_new_name [id=da7daf89e87b4602824738ea0b6ac070]) successfully deleted
I20240501 19:57:03.301218 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 05c2d3abee2e4610bed4ee58e9153df5 (table rebalancer_dynamic_table_new_name [id=da7daf89e87b4602824738ea0b6ac070]) successfully deleted
I20240501 19:57:03.327445 27425 consensus_queue.cc:237] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:03.329025 27032 raft_consensus.cc:1261] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:03.329500 1023 consensus_queue.cc:1035] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:03.331627 929 raft_consensus.cc:2932] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:03.332950 24594 raft_consensus.cc:1261] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:03.333364 929 consensus_queue.cc:1035] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:03.334661 27032 raft_consensus.cc:2932] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:03.334669 24594 raft_consensus.cc:2932] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:03.335678 24450 catalog_manager.cc:5484] T fce9f9c2a5be47d3986e3d7b53570b12 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:03.339958 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet fce9f9c2a5be47d3986e3d7b53570b12 with delete_type TABLET_DATA_TOMBSTONED (TS 53e34bb09d8e41129a5683079ff63610 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:03.340210 1294 tablet_replica.cc:330] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:03.340404 1294 raft_consensus.cc:2229] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:03.340562 1294 raft_consensus.cc:2258] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:03.341020 1294 ts_tablet_manager.cc:1875] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:03.343138 1294 ts_tablet_manager.cc:1888] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:03.343207 1294 log.cc:1196] T fce9f9c2a5be47d3986e3d7b53570b12 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/fce9f9c2a5be47d3986e3d7b53570b12
I20240501 19:57:03.343910 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet fce9f9c2a5be47d3986e3d7b53570b12 (table rebalancer_extra_table_6 [id=158598ed5c394493809771889676a083]) successfully deleted
I20240501 19:57:03.380187 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:03.386294 27143 tablet_service.cc:1467] Processing CreateTablet for tablet e3b90515b8144ce991d834087705fc5c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc7c4ca559f34147a10007780bae4bb9]), partition=
I20240501 19:57:03.386592 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b90515b8144ce991d834087705fc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.386567 24837 tablet_service.cc:1467] Processing CreateTablet for tablet e3b90515b8144ce991d834087705fc5c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc7c4ca559f34147a10007780bae4bb9]), partition=
I20240501 19:57:03.386745 27011 tablet_service.cc:1467] Processing CreateTablet for tablet e3b90515b8144ce991d834087705fc5c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc7c4ca559f34147a10007780bae4bb9]), partition=
I20240501 19:57:03.386801 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b90515b8144ce991d834087705fc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.386982 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b90515b8144ce991d834087705fc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.390362 1339 tablet_bootstrap.cc:492] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:03.391925 1339 tablet_bootstrap.cc:654] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.393054 1339 tablet_bootstrap.cc:492] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:03.393146 1339 ts_tablet_manager.cc:1367] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:03.393604 1339 raft_consensus.cc:348] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.393715 1339 raft_consensus.cc:374] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.393752 1339 raft_consensus.cc:729] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.393887 1339 consensus_queue.cc:260] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.394158 1339 ts_tablet_manager.cc:1398] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:03.396946 1338 tablet_bootstrap.cc:492] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:03.397907 1338 tablet_bootstrap.cc:654] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.399089 1338 tablet_bootstrap.cc:492] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:03.399215 1338 ts_tablet_manager.cc:1367] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:03.399688 1338 raft_consensus.cc:348] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.399840 1338 raft_consensus.cc:374] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.399904 1338 raft_consensus.cc:729] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.400074 1338 consensus_queue.cc:260] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.400382 1338 ts_tablet_manager.cc:1398] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:03.412047 1340 tablet_bootstrap.cc:492] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:03.413267 1340 tablet_bootstrap.cc:654] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.423388 1340 tablet_bootstrap.cc:492] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:03.423520 1340 ts_tablet_manager.cc:1367] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:03.424609 1340 raft_consensus.cc:348] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.424778 1340 raft_consensus.cc:374] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.424852 1340 raft_consensus.cc:729] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.425014 1340 consensus_queue.cc:260] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.425354 1340 ts_tablet_manager.cc:1398] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:03.471542 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:03.477331 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 4716f9a3e62a495da309ad14614fceb6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.477458 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 4716f9a3e62a495da309ad14614fceb6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.477715 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4716f9a3e62a495da309ad14614fceb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.477787 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4716f9a3e62a495da309ad14614fceb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.479583 1363 tablet_bootstrap.cc:492] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:03.480293 27267 tablet_service.cc:1467] Processing CreateTablet for tablet c76585af218d4ebea389db62aa99a3f7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.480351 27405 tablet_service.cc:1467] Processing CreateTablet for tablet c76585af218d4ebea389db62aa99a3f7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.480576 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76585af218d4ebea389db62aa99a3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.480597 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76585af218d4ebea389db62aa99a3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.480582 1363 tablet_bootstrap.cc:654] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.481729 1363 tablet_bootstrap.cc:492] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:03.481807 1363 ts_tablet_manager.cc:1367] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:03.482263 1363 raft_consensus.cc:348] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.482365 1363 raft_consensus.cc:374] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.482395 1363 raft_consensus.cc:729] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.482528 1365 tablet_bootstrap.cc:492] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:03.482547 1338 tablet_bootstrap.cc:492] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:03.482556 1363 consensus_queue.cc:260] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.482841 1363 ts_tablet_manager.cc:1398] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:03.483630 1338 tablet_bootstrap.cc:654] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.483764 1365 tablet_bootstrap.cc:654] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.484493 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 244a44d25272441c939d6d69644eaf56 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.484719 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244a44d25272441c939d6d69644eaf56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.486811 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 4716f9a3e62a495da309ad14614fceb6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.487107 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4716f9a3e62a495da309ad14614fceb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.487167 1338 tablet_bootstrap.cc:492] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:03.487237 1338 ts_tablet_manager.cc:1367] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:03.488551 1365 tablet_bootstrap.cc:492] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:03.487670 1338 raft_consensus.cc:348] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.488710 1338 raft_consensus.cc:374] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.488750 1338 raft_consensus.cc:729] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.488869 1338 consensus_queue.cc:260] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.489145 1338 ts_tablet_manager.cc:1398] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:03.489552 1338 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:03.489651 1340 tablet_bootstrap.cc:492] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:03.489674 24705 tablet_service.cc:1467] Processing CreateTablet for tablet c76585af218d4ebea389db62aa99a3f7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.489960 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76585af218d4ebea389db62aa99a3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.490473 1340 tablet_bootstrap.cc:654] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.490504 1338 tablet_bootstrap.cc:654] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.491304 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 244a44d25272441c939d6d69644eaf56 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.491310 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 244a44d25272441c939d6d69644eaf56 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=98633641d1334d00b792610a6ff59753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:03.491421 1340 tablet_bootstrap.cc:492] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:03.491490 1340 ts_tablet_manager.cc:1367] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:03.491530 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244a44d25272441c939d6d69644eaf56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.491537 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244a44d25272441c939d6d69644eaf56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:03.491936 1340 raft_consensus.cc:348] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.492022 1340 raft_consensus.cc:374] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.492049 1340 raft_consensus.cc:729] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.492182 1340 consensus_queue.cc:260] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.492462 1340 ts_tablet_manager.cc:1398] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:03.493570 1371 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:03.494537 1371 tablet_bootstrap.cc:654] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.488631 1365 ts_tablet_manager.cc:1367] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:03.495357 1365 raft_consensus.cc:348] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.495504 1365 raft_consensus.cc:374] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.495554 1365 raft_consensus.cc:729] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.495679 1365 consensus_queue.cc:260] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.495951 1365 ts_tablet_manager.cc:1398] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:03.496239 1370 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:03.497170 1370 tablet_bootstrap.cc:654] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.498551 1363 tablet_bootstrap.cc:492] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:03.499781 1363 tablet_bootstrap.cc:654] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.500989 1363 tablet_bootstrap.cc:492] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:03.501072 1363 ts_tablet_manager.cc:1367] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:57:03.501533 1363 raft_consensus.cc:348] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.501636 1363 raft_consensus.cc:374] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.501670 1363 raft_consensus.cc:729] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.501782 1363 consensus_queue.cc:260] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.502048 1363 ts_tablet_manager.cc:1398] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:03.502099 1370 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:03.502177 1370 ts_tablet_manager.cc:1367] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:03.502756 1370 raft_consensus.cc:348] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.502875 1370 raft_consensus.cc:374] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.502912 1370 raft_consensus.cc:729] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.502966 1338 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:03.503023 1370 consensus_queue.cc:260] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.503162 1338 ts_tablet_manager.cc:1367] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:57:03.503628 1338 raft_consensus.cc:348] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.503726 1338 raft_consensus.cc:374] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.503760 1338 raft_consensus.cc:729] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.503871 1338 consensus_queue.cc:260] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.503985 1371 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:03.504092 1371 ts_tablet_manager.cc:1367] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240501 19:57:03.504130 1338 ts_tablet_manager.cc:1398] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:03.504451 1370 ts_tablet_manager.cc:1398] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:03.504608 1370 tablet_bootstrap.cc:492] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:03.504549 1371 raft_consensus.cc:348] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.504644 1371 raft_consensus.cc:374] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.504689 1371 raft_consensus.cc:729] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.504801 1371 consensus_queue.cc:260] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.505640 1023 consensus_queue.cc:1035] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.507021 1370 tablet_bootstrap.cc:654] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:03.511212 929 consensus_queue.cc:1035] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.512898 1371 ts_tablet_manager.cc:1398] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240501 19:57:03.513955 1370 tablet_bootstrap.cc:492] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:03.514117 1370 ts_tablet_manager.cc:1367] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 19:57:03.514511 1370 raft_consensus.cc:348] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.514595 1370 raft_consensus.cc:374] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:03.514617 1370 raft_consensus.cc:729] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:03.514748 1370 consensus_queue.cc:260] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.515004 1370 ts_tablet_manager.cc:1398] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:03.533669 561 raft_consensus.cc:482] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:03.533784 561 raft_consensus.cc:504] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.534132 561 leader_election.cc:288] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:03.534510 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b90515b8144ce991d834087705fc5c" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:03.534509 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b90515b8144ce991d834087705fc5c" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:03.534653 27032 raft_consensus.cc:2454] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:03.534653 24856 raft_consensus.cc:2454] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:03.534932 27096 leader_election.cc:302] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:57:03.535184 561 raft_consensus.cc:2781] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:03.535248 561 raft_consensus.cc:482] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:03.535293 561 raft_consensus.cc:3037] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.536252 561 raft_consensus.cc:504] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.536543 561 leader_election.cc:288] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:03.536805 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b90515b8144ce991d834087705fc5c" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:03.536895 24856 raft_consensus.cc:3037] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.537820 24856 raft_consensus.cc:2454] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:03.538070 27097 leader_election.cc:302] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:03.538158 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b90515b8144ce991d834087705fc5c" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:03.538280 27032 raft_consensus.cc:3037] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.538282 561 raft_consensus.cc:2781] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:03.538357 561 raft_consensus.cc:686] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:03.538512 561 consensus_queue.cc:237] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.539291 27032 raft_consensus.cc:2454] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:03.541008 24446 catalog_manager.cc:5484] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:03.548452 32319 raft_consensus.cc:482] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:03.548564 32319 raft_consensus.cc:504] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.549010 32319 leader_election.cc:288] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:03.549506 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244a44d25272441c939d6d69644eaf56" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:03.549633 24725 raft_consensus.cc:2454] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:03.549970 24529 leader_election.cc:302] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:03.550019 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244a44d25272441c939d6d69644eaf56" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:03.550130 27162 raft_consensus.cc:2454] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:03.550153 32319 raft_consensus.cc:2781] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:03.550204 32319 raft_consensus.cc:482] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:03.550256 32319 raft_consensus.cc:3037] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.551450 32319 raft_consensus.cc:504] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.551800 32319 leader_election.cc:288] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:03.552074 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244a44d25272441c939d6d69644eaf56" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:03.552196 24725 raft_consensus.cc:3037] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.552197 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244a44d25272441c939d6d69644eaf56" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:03.552302 27162 raft_consensus.cc:3037] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.553145 27162 raft_consensus.cc:2454] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:03.553167 24725 raft_consensus.cc:2454] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:03.553381 24531 leader_election.cc:302] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:03.553550 32319 raft_consensus.cc:2781] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:03.553638 32319 raft_consensus.cc:686] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:03.553783 32319 consensus_queue.cc:237] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:03.556068 24446 catalog_manager.cc:5484] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:03.567118 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:03.567921 24574 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:03.572443 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:03.573108 24966 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:03.574851 27267 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:03.595177 27405 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:03.606191 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:03.611590 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:03.642222 561 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:03.642339 561 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.642786 561 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:03.643093 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716f9a3e62a495da309ad14614fceb6" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:03.643260 27425 raft_consensus.cc:2454] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:03.643275 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716f9a3e62a495da309ad14614fceb6" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:03.643383 27032 raft_consensus.cc:2454] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:03.643582 27098 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:03.643771 561 raft_consensus.cc:2781] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:03.643832 561 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:03.643867 561 raft_consensus.cc:3037] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.644982 561 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.645290 561 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:03.645692 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716f9a3e62a495da309ad14614fceb6" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:03.645788 27032 raft_consensus.cc:3037] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.646065 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716f9a3e62a495da309ad14614fceb6" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:03.646157 27425 raft_consensus.cc:3037] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.646814 27032 raft_consensus.cc:2454] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:03.647105 27096 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:57:03.647264 561 raft_consensus.cc:2781] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:03.647266 27425 raft_consensus.cc:2454] T 4716f9a3e62a495da309ad14614fceb6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:03.647390 561 raft_consensus.cc:686] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:03.647529 561 consensus_queue.cc:237] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:03.650326 24446 catalog_manager.cc:5484] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:03.652935 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:03.658110 27032 raft_consensus.cc:1261] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:03.658268 24856 raft_consensus.cc:1261] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:03.658756 1200 consensus_queue.cc:1035] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.658880 561 consensus_queue.cc:1035] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.661525 1397 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:03.661896 1397 tablet.cc:1620] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:03.662349 24446 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet e3b90515b8144ce991d834087705fc5c (table rebalancer_dynamic_table [id=dc7c4ca559f34147a10007780bae4bb9]). Expected version 1 got 0
I20240501 19:57:03.662470 1398 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:03.662755 1398 tablet.cc:1620] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:03.665639 24450 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet e3b90515b8144ce991d834087705fc5c (table rebalancer_dynamic_table [id=dc7c4ca559f34147a10007780bae4bb9]). Expected version 1 got 0
I20240501 19:57:03.668907 24453 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet e3b90515b8144ce991d834087705fc5c (table rebalancer_dynamic_table [id=dc7c4ca559f34147a10007780bae4bb9]). Expected version 1 got 0
I20240501 19:57:03.667158 1399 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:03.669664 1399 tablet.cc:1620] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:03.669939 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=dc7c4ca559f34147a10007780bae4bb9] alter complete (version 1)
I20240501 19:57:03.676553 27163 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc"
dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51864
I20240501 19:57:03.676707 27163 raft_consensus.cc:595] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Received request to transfer leadership
W20240501 19:57:03.725035 24774 tablet.cc:2369] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:03.764106 561 raft_consensus.cc:979] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: : Instructing follower 4a6e54e86a9949e5970b42e8e32e7561 to start an election
I20240501 19:57:03.764201 561 raft_consensus.cc:1067] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Signalling peer 4a6e54e86a9949e5970b42e8e32e7561 to start an election
I20240501 19:57:03.764413 1023 raft_consensus.cc:482] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:03.764515 1023 raft_consensus.cc:504] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.764915 27425 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc"
dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
from {username='slave'} at 127.17.65.6:43989
I20240501 19:57:03.765033 27425 raft_consensus.cc:482] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:03.765076 1023 leader_election.cc:288] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:03.765086 27425 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:03.765417 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76585af218d4ebea389db62aa99a3f7" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:03.765550 24725 raft_consensus.cc:2454] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:03.765897 27359 leader_election.cc:302] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:03.766075 1023 raft_consensus.cc:2781] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:03.766127 1023 raft_consensus.cc:482] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:03.766173 1023 raft_consensus.cc:3037] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.766358 27425 raft_consensus.cc:504] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:03.766814 27425 leader_election.cc:288] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 2 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:03.767165 1023 raft_consensus.cc:504] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.767215 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76585af218d4ebea389db62aa99a3f7" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:03.767254 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:03.767334 27276 raft_consensus.cc:2454] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:03.767387 27278 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:03.767490 1023 leader_election.cc:288] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:03.768000 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:03.768066 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:03.768112 27163 raft_consensus.cc:3032] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:03.768149 27163 raft_consensus.cc:729] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:03.768169 24856 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:03.768353 27278 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 2.
I20240501 19:57:03.768307 27163 consensus_queue.cc:260] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:03.768498 27163 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:03.769191 24856 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 2.
I20240501 19:57:03.769477 27163 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 2.
I20240501 19:57:03.770009 27361 leader_election.cc:302] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:03.770504 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76585af218d4ebea389db62aa99a3f7" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:03.770643 24725 raft_consensus.cc:3037] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.771692 24725 raft_consensus.cc:2454] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:03.772480 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76585af218d4ebea389db62aa99a3f7" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:03.772589 27276 raft_consensus.cc:3037] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:03.772541 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:03.773541 27276 raft_consensus.cc:2454] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:03.773985 27361 leader_election.cc:302] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:03.774176 929 raft_consensus.cc:2781] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:03.774338 929 raft_consensus.cc:686] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:57:03.774477 929 consensus_queue.cc:237] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:03.775225 929 raft_consensus.cc:2781] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:03.775296 929 raft_consensus.cc:686] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:57:03.775403 929 consensus_queue.cc:237] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:03.777791 24449 catalog_manager.cc:5484] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 1 to 2, leader changed from 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 2 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:03.778041 24449 catalog_manager.cc:5484] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:03.780339 1397 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:03.780635 1397 tablet.cc:1620] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:03.781185 1399 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:03.781487 1399 tablet.cc:1620] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:03.781570 1398 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:03.781860 1398 tablet.cc:1620] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:03.787321 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=dc7c4ca559f34147a10007780bae4bb9] alter complete (version 2)
I20240501 19:57:03.802174 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:03.816042 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:03.820976 27267 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:03.823212 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:03.827536 24966 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:03.829258 24574 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:03.829849 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:03.860116 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:03.889655 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:03.894953 1397 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:03.895768 1398 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:03.895807 1399 tablet.cc:1722] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:03.903584 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=dc7c4ca559f34147a10007780bae4bb9] alter complete (version 3)
I20240501 19:57:03.975842 32319 consensus_queue.cc:1035] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.977077 32319 consensus_queue.cc:1035] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:03.995836 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:03.999984 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:04.004107 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:04.006371 24447 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:04.006521 24447 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:04.009524 24447 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet e3b90515b8144ce991d834087705fc5c
I20240501 19:57:04.010286 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b90515b8144ce991d834087705fc5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:04 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:04.010547 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b90515b8144ce991d834087705fc5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:04 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:04.010715 1512 tablet_replica.cc:330] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:04.010838 1512 raft_consensus.cc:2229] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:04.011077 1512 raft_consensus.cc:2258] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:04.011782 1512 ts_tablet_manager.cc:1875] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:04.013463 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b90515b8144ce991d834087705fc5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:04 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:04.014071 1512 ts_tablet_manager.cc:1888] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:04.014189 1512 log.cc:1196] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/e3b90515b8144ce991d834087705fc5c
I20240501 19:57:04.014626 1512 ts_tablet_manager.cc:1909] T e3b90515b8144ce991d834087705fc5c P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:57:04.015482 1513 tablet_replica.cc:330] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:04.015615 1513 raft_consensus.cc:2229] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:04.015795 1513 raft_consensus.cc:2258] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:04.016140 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:04.016623 1513 ts_tablet_manager.cc:1875] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:04.018620 1513 ts_tablet_manager.cc:1888] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:04.018699 1513 log.cc:1196] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/e3b90515b8144ce991d834087705fc5c
I20240501 19:57:04.018931 1513 ts_tablet_manager.cc:1909] T e3b90515b8144ce991d834087705fc5c P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:57:04.015998 1514 tablet_replica.cc:330] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:04.019512 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet e3b90515b8144ce991d834087705fc5c (table rebalancer_dynamic_table_new_name [id=dc7c4ca559f34147a10007780bae4bb9]) successfully deleted
I20240501 19:57:04.019603 1514 raft_consensus.cc:2229] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:04.020705 1514 raft_consensus.cc:2258] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:04.021746 1514 ts_tablet_manager.cc:1875] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:04.025327 1514 ts_tablet_manager.cc:1888] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:04.025393 1514 log.cc:1196] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/e3b90515b8144ce991d834087705fc5c
I20240501 19:57:04.025681 1514 ts_tablet_manager.cc:1909] T e3b90515b8144ce991d834087705fc5c P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:04.026361 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet e3b90515b8144ce991d834087705fc5c (table rebalancer_dynamic_table_new_name [id=dc7c4ca559f34147a10007780bae4bb9]) successfully deleted
I20240501 19:57:04.027213 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet e3b90515b8144ce991d834087705fc5c (table rebalancer_dynamic_table_new_name [id=dc7c4ca559f34147a10007780bae4bb9]) successfully deleted
I20240501 19:57:04.027794 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:04.038128 27267 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:04.043043 24574 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:04.056169 24966 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:04.057546 1466 consensus_queue.cc:1035] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.074827 1466 consensus_queue.cc:1035] T 4716f9a3e62a495da309ad14614fceb6 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.098990 24594 consensus_queue.cc:237] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:04.100162 27425 raft_consensus.cc:1261] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:04.100203 24856 raft_consensus.cc:1261] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:04.100598 32319 consensus_queue.cc:1035] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:04.100795 1202 consensus_queue.cc:1035] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:04.102591 1536 raft_consensus.cc:2932] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:04.103003 27425 raft_consensus.cc:2932] T 2e30faedce3742e0ba83de21059613d1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:04.103013 24856 raft_consensus.cc:2932] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:04.106081 24447 catalog_manager.cc:5484] T 2e30faedce3742e0ba83de21059613d1 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:04.109190 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e30faedce3742e0ba83de21059613d1 with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:04.110203 1537 tablet_replica.cc:330] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:04.110375 1537 raft_consensus.cc:2229] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:04.110698 1537 raft_consensus.cc:2258] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:04.111294 1537 ts_tablet_manager.cc:1875] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:04.112478 24449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:04.117897 1537 ts_tablet_manager.cc:1888] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:04.118039 1537 log.cc:1196] T 2e30faedce3742e0ba83de21059613d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/2e30faedce3742e0ba83de21059613d1
I20240501 19:57:04.118831 24837 tablet_service.cc:1467] Processing CreateTablet for tablet ff8157a5f1604dc9b9267fe330ebe1e3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5da51dcf11ea4f6993ae5ef6fb739f92]), partition=
I20240501 19:57:04.118825 27267 tablet_service.cc:1467] Processing CreateTablet for tablet ff8157a5f1604dc9b9267fe330ebe1e3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5da51dcf11ea4f6993ae5ef6fb739f92]), partition=
I20240501 19:57:04.119093 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8157a5f1604dc9b9267fe330ebe1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.119105 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8157a5f1604dc9b9267fe330ebe1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.121598 24574 tablet_service.cc:1467] Processing CreateTablet for tablet ff8157a5f1604dc9b9267fe330ebe1e3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5da51dcf11ea4f6993ae5ef6fb739f92]), partition=
I20240501 19:57:04.121819 1548 tablet_bootstrap.cc:492] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:04.121850 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8157a5f1604dc9b9267fe330ebe1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.122923 1548 tablet_bootstrap.cc:654] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.124320 1552 tablet_bootstrap.cc:492] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:04.124612 1548 tablet_bootstrap.cc:492] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:04.125312 1552 tablet_bootstrap.cc:654] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.124732 1548 ts_tablet_manager.cc:1367] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:04.127015 1548 raft_consensus.cc:348] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.127246 1548 raft_consensus.cc:374] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.127316 1548 raft_consensus.cc:729] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.127486 1548 consensus_queue.cc:260] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.127795 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 2e30faedce3742e0ba83de21059613d1 (table rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]) successfully deleted
I20240501 19:57:04.128299 1548 ts_tablet_manager.cc:1398] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:04.128317 1549 tablet_bootstrap.cc:492] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:04.129179 1549 tablet_bootstrap.cc:654] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.131484 1552 tablet_bootstrap.cc:492] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:04.132202 1549 tablet_bootstrap.cc:492] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:04.132385 1549 ts_tablet_manager.cc:1367] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:04.133008 1552 ts_tablet_manager.cc:1367] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 19:57:04.133116 1549 raft_consensus.cc:348] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.133531 1549 raft_consensus.cc:374] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.133500 1552 raft_consensus.cc:348] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.133605 1549 raft_consensus.cc:729] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.133610 1552 raft_consensus.cc:374] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.133656 1552 raft_consensus.cc:729] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.133805 1552 consensus_queue.cc:260] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.133813 1549 consensus_queue.cc:260] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.134342 1552 ts_tablet_manager.cc:1398] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:04.139503 1549 ts_tablet_manager.cc:1398] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:04.176946 1441 consensus_queue.cc:1035] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.191112 929 consensus_queue.cc:1035] T c76585af218d4ebea389db62aa99a3f7 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.217543 27163 raft_consensus.cc:1261] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:04.218358 929 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:04.245473 1373 raft_consensus.cc:482] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:04.245608 1373 raft_consensus.cc:504] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.246037 1373 leader_election.cc:288] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:04.246403 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8157a5f1604dc9b9267fe330ebe1e3" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:04.246349 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8157a5f1604dc9b9267fe330ebe1e3" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:04.246598 24857 raft_consensus.cc:2454] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:04.246642 24594 raft_consensus.cc:2454] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:04.246903 27228 leader_election.cc:302] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:04.247189 1373 raft_consensus.cc:2781] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:04.247246 1373 raft_consensus.cc:482] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:04.247274 1373 raft_consensus.cc:3037] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.248199 1373 raft_consensus.cc:504] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.248536 1373 leader_election.cc:288] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:04.248940 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8157a5f1604dc9b9267fe330ebe1e3" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:04.249059 24857 raft_consensus.cc:3037] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.249619 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8157a5f1604dc9b9267fe330ebe1e3" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:04.249709 24594 raft_consensus.cc:3037] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.249989 24857 raft_consensus.cc:2454] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:04.250241 27228 leader_election.cc:302] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:04.250380 1373 raft_consensus.cc:2781] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:04.250460 1373 raft_consensus.cc:686] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:04.250582 24594 raft_consensus.cc:2454] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:04.250589 1373 consensus_queue.cc:237] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:04.252449 24448 catalog_manager.cc:5484] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:04.273550 24837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:04.273644 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:04.275763 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:04.277637 27011 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:04.278725 27267 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:04.282126 24966 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:04.295604 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:04.298059 24574 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:04.309973 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:04.323197 27278 raft_consensus.cc:1261] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:04.323302 27267 tablet_service.cc:1467] Processing CreateTablet for tablet c6753d5ac6144c1ba71bc30474f89be7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.323601 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6753d5ac6144c1ba71bc30474f89be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.323714 929 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:04.325706 1549 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:04.325953 27143 tablet_service.cc:1467] Processing CreateTablet for tablet c6753d5ac6144c1ba71bc30474f89be7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.326201 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6753d5ac6144c1ba71bc30474f89be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.326706 1549 tablet_bootstrap.cc:654] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.329226 1606 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:04.331513 24966 tablet_service.cc:1467] Processing CreateTablet for tablet c6753d5ac6144c1ba71bc30474f89be7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.331650 27010 tablet_service.cc:1467] Processing CreateTablet for tablet 2724041871e04cfdacb91ba5b4026780 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.331810 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6753d5ac6144c1ba71bc30474f89be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.331907 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2724041871e04cfdacb91ba5b4026780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.331652 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0bf1ddc7aa4a5ea4a7a863337e40d1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.332129 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0bf1ddc7aa4a5ea4a7a863337e40d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.334934 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0bf1ddc7aa4a5ea4a7a863337e40d1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.335307 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0bf1ddc7aa4a5ea4a7a863337e40d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.336362 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0bf1ddc7aa4a5ea4a7a863337e40d1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.336611 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0bf1ddc7aa4a5ea4a7a863337e40d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.338315 1548 tablet_bootstrap.cc:492] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:04.338522 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 2724041871e04cfdacb91ba5b4026780 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.338764 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2724041871e04cfdacb91ba5b4026780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.334934 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 2724041871e04cfdacb91ba5b4026780 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:04.340243 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2724041871e04cfdacb91ba5b4026780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.343576 1548 tablet_bootstrap.cc:654] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.344586 1548 tablet_bootstrap.cc:492] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:04.344679 1548 ts_tablet_manager.cc:1367] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240501 19:57:04.345029 1548 raft_consensus.cc:348] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.345106 1548 raft_consensus.cc:374] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.345126 1548 raft_consensus.cc:729] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.345234 1548 consensus_queue.cc:260] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.345515 1548 ts_tablet_manager.cc:1398] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:04.345679 1548 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:04.346592 1548 tablet_bootstrap.cc:654] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.348453 1606 tablet_bootstrap.cc:654] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.348704 24857 raft_consensus.cc:1261] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:04.349645 929 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:04.350371 1548 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:04.350476 1548 ts_tablet_manager.cc:1367] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:57:04.350919 1548 raft_consensus.cc:348] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.351024 1548 raft_consensus.cc:374] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.351109 1548 raft_consensus.cc:729] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.351274 1548 consensus_queue.cc:260] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.351536 1548 ts_tablet_manager.cc:1398] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.353175 1607 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:04.354390 1607 tablet_bootstrap.cc:654] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.355214 1608 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:04.356405 1608 tablet_bootstrap.cc:654] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.357282 1606 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:04.357375 1606 ts_tablet_manager.cc:1367] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240501 19:57:04.357775 1606 raft_consensus.cc:348] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.357867 1606 raft_consensus.cc:374] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.357900 1606 raft_consensus.cc:729] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.358031 1606 consensus_queue.cc:260] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.358294 1606 ts_tablet_manager.cc:1398] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:04.358536 1606 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:04.359050 1549 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:04.359153 1549 ts_tablet_manager.cc:1367] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.002s
I20240501 19:57:04.359651 1549 raft_consensus.cc:348] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.359763 1549 raft_consensus.cc:374] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.359798 1549 raft_consensus.cc:729] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.359902 1606 tablet_bootstrap.cc:654] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.359915 1549 consensus_queue.cc:260] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.360193 1549 ts_tablet_manager.cc:1398] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:04.361037 1608 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:04.361047 1606 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:04.361122 1608 ts_tablet_manager.cc:1367] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:04.361124 1606 ts_tablet_manager.cc:1367] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:04.361747 1606 raft_consensus.cc:348] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.361783 1608 raft_consensus.cc:348] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.361866 1606 raft_consensus.cc:374] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.361897 1606 raft_consensus.cc:729] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.361924 1608 raft_consensus.cc:374] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.361979 1608 raft_consensus.cc:729] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.362008 1606 consensus_queue.cc:260] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.362103 1608 consensus_queue.cc:260] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.362476 1608 ts_tablet_manager.cc:1398] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.362957 1608 tablet_bootstrap.cc:492] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:04.363070 1606 ts_tablet_manager.cc:1398] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:04.363942 1608 tablet_bootstrap.cc:654] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.364652 1607 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:04.364768 1607 ts_tablet_manager.cc:1367] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:04.365206 1608 tablet_bootstrap.cc:492] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:04.365306 1608 ts_tablet_manager.cc:1367] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:04.365337 1607 raft_consensus.cc:348] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.365723 1608 raft_consensus.cc:348] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.365782 1607 raft_consensus.cc:374] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.365813 1608 raft_consensus.cc:374] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.365828 1607 raft_consensus.cc:729] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.365841 1608 raft_consensus.cc:729] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.365943 1608 consensus_queue.cc:260] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.365948 1607 consensus_queue.cc:260] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.366184 1608 ts_tablet_manager.cc:1398] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.366235 1607 ts_tablet_manager.cc:1398] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.367252 1607 tablet_bootstrap.cc:492] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:04.368306 1607 tablet_bootstrap.cc:654] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.370046 1607 tablet_bootstrap.cc:492] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:04.370182 1607 ts_tablet_manager.cc:1367] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:04.370707 1607 raft_consensus.cc:348] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.370858 1607 raft_consensus.cc:374] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.370931 1607 raft_consensus.cc:729] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.371146 1607 consensus_queue.cc:260] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.371490 1607 ts_tablet_manager.cc:1398] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.374316 24727 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d"
dest_uuid: "9081292597be4a72a42b216fddcd7971"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41914
I20240501 19:57:04.374433 24727 raft_consensus.cc:595] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:04.387413 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:04.392222 24857 raft_consensus.cc:1261] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:04.392591 24594 raft_consensus.cc:1261] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:04.392976 1038 consensus_queue.cc:1035] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.393249 1038 consensus_queue.cc:1035] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.396198 24448 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet ff8157a5f1604dc9b9267fe330ebe1e3 (table rebalancer_dynamic_table [id=5da51dcf11ea4f6993ae5ef6fb739f92]). Expected version 1 got 0
I20240501 19:57:04.398049 1631 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:04.399152 24448 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet ff8157a5f1604dc9b9267fe330ebe1e3 (table rebalancer_dynamic_table [id=5da51dcf11ea4f6993ae5ef6fb739f92]). Expected version 1 got 0
I20240501 19:57:04.400512 1631 tablet.cc:1620] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:04.401259 1633 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:04.401300 1293 mvcc.cc:205] Tried to move back new op lower bound from 7022974666306117632 to 7022974665731395584. Current Snapshot: MvccSnapshot[applied={T|T < 7022974666306117632}]
I20240501 19:57:04.401679 1633 tablet.cc:1620] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:04.402678 1632 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:04.405210 24448 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet ff8157a5f1604dc9b9267fe330ebe1e3 (table rebalancer_dynamic_table [id=5da51dcf11ea4f6993ae5ef6fb739f92]). Expected version 1 got 0
I20240501 19:57:04.406220 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=5da51dcf11ea4f6993ae5ef6fb739f92] alter complete (version 1)
I20240501 19:57:04.403185 1632 tablet.cc:1620] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:04.439159 925 raft_consensus.cc:482] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:04.439285 925 raft_consensus.cc:504] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.439707 925 leader_election.cc:288] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:04.440156 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2724041871e04cfdacb91ba5b4026780" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:04.440294 27162 raft_consensus.cc:2454] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:04.440296 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2724041871e04cfdacb91ba5b4026780" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:04.440446 27031 raft_consensus.cc:2454] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:04.440788 24793 leader_election.cc:302] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:04.441051 925 raft_consensus.cc:2781] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:04.441112 925 raft_consensus.cc:482] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:04.441148 925 raft_consensus.cc:3037] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.442258 925 raft_consensus.cc:504] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.442579 925 leader_election.cc:288] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:04.442943 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2724041871e04cfdacb91ba5b4026780" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:04.443066 27031 raft_consensus.cc:3037] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.443260 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2724041871e04cfdacb91ba5b4026780" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:04.443370 27162 raft_consensus.cc:3037] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.444147 27031 raft_consensus.cc:2454] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:04.444422 27162 raft_consensus.cc:2454] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:04.445174 24791 leader_election.cc:302] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:04.445364 925 raft_consensus.cc:2781] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:04.445560 925 raft_consensus.cc:686] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:04.445719 925 consensus_queue.cc:237] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.449813 24450 catalog_manager.cc:5484] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:04.473982 1666 raft_consensus.cc:482] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:04.475160 1666 raft_consensus.cc:504] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.476140 1666 leader_election.cc:288] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:04.476543 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0bf1ddc7aa4a5ea4a7a863337e40d1" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:04.476703 24857 raft_consensus.cc:2454] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:04.476663 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0bf1ddc7aa4a5ea4a7a863337e40d1" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:04.476807 24987 raft_consensus.cc:2454] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:04.478034 26966 leader_election.cc:302] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:04.481271 1666 raft_consensus.cc:2781] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:04.481338 1666 raft_consensus.cc:482] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:04.481386 1666 raft_consensus.cc:3037] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.482650 1666 raft_consensus.cc:504] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.483260 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0bf1ddc7aa4a5ea4a7a863337e40d1" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:04.483386 24857 raft_consensus.cc:3037] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.484006 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0bf1ddc7aa4a5ea4a7a863337e40d1" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:04.484117 24987 raft_consensus.cc:3037] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.484449 24857 raft_consensus.cc:2454] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:04.484743 26966 leader_election.cc:302] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:04.485045 1665 raft_consensus.cc:2781] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:04.485200 24987 raft_consensus.cc:2454] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:04.485219 1665 raft_consensus.cc:686] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:04.485365 1665 consensus_queue.cc:237] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.488961 1666 leader_election.cc:288] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:04.489639 24453 catalog_manager.cc:5484] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:04.512364 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:04.517283 1373 raft_consensus.cc:482] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:04.517398 1373 raft_consensus.cc:504] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.517897 1373 leader_election.cc:288] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:04.518034 1631 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:04.518342 1631 tablet.cc:1620] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:04.518653 1632 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:04.518954 1632 tablet.cc:1620] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:04.519429 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6753d5ac6144c1ba71bc30474f89be7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:04.519569 27162 raft_consensus.cc:2454] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:04.519909 27229 leader_election.cc:302] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:04.521749 1290 raft_consensus.cc:979] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: : Instructing follower b9fd7c49d12b48a7ae73ab09876a0b82 to start an election
I20240501 19:57:04.521831 1290 raft_consensus.cc:1067] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Signalling peer b9fd7c49d12b48a7ae73ab09876a0b82 to start an election
I20240501 19:57:04.528442 1633 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:04.528522 1670 raft_consensus.cc:2781] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:04.528579 1670 raft_consensus.cc:482] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:04.528641 1670 raft_consensus.cc:3037] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.528743 1633 tablet.cc:1620] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:04.529263 24986 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d"
dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
from {username='slave'} at 127.17.65.2:54941
I20240501 19:57:04.529433 24986 raft_consensus.cc:482] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:04.529503 24986 raft_consensus.cc:3037] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:04.529567 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=5da51dcf11ea4f6993ae5ef6fb739f92] alter complete (version 2)
I20240501 19:57:04.529597 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6753d5ac6144c1ba71bc30474f89be7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:04.529701 24987 raft_consensus.cc:2454] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:04.530526 1670 raft_consensus.cc:504] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.530561 24986 raft_consensus.cc:504] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.530844 1670 leader_election.cc:288] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:04.531147 24986 leader_election.cc:288] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:04.531172 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6753d5ac6144c1ba71bc30474f89be7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:04.531267 24987 raft_consensus.cc:3037] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.531221 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6753d5ac6144c1ba71bc30474f89be7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:04.531314 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:04.531366 27162 raft_consensus.cc:3037] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.531427 24856 raft_consensus.cc:3037] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:04.531647 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:04.531759 27031 raft_consensus.cc:3037] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:04.532177 24987 raft_consensus.cc:2454] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:04.532335 27162 raft_consensus.cc:2454] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:04.532413 24856 raft_consensus.cc:2454] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 2.
I20240501 19:57:04.532521 27228 leader_election.cc:302] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:04.532699 1670 raft_consensus.cc:2781] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:04.532783 1670 raft_consensus.cc:686] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:04.532789 27031 raft_consensus.cc:2454] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 2.
I20240501 19:57:04.533001 1670 consensus_queue.cc:237] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:04.533099 24922 leader_election.cc:302] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:04.533269 1562 raft_consensus.cc:2781] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:04.533357 1562 raft_consensus.cc:686] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:04.533466 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1f7372e5ce42b284cf1179c72f578d" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:04.533512 1562 consensus_queue.cc:237] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.533576 24727 raft_consensus.cc:3032] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:04.533617 24727 raft_consensus.cc:729] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:04.533759 24727 consensus_queue.cc:260] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:04.533934 24727 raft_consensus.cc:3037] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:04.534871 24727 raft_consensus.cc:2454] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 2.
I20240501 19:57:04.538117 24453 catalog_manager.cc:5484] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 1 to 2, leader changed from 9081292597be4a72a42b216fddcd7971 (127.17.65.2) to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 2 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:04.540308 24450 catalog_manager.cc:5484] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:04.560266 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:04.567265 24836 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:04.577963 24574 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:04.581275 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:04.586208 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:04.586906 27267 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:04.588300 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:04.611348 27011 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:04.631425 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:04.639847 1631 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:04.642304 1633 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:04.648389 1632 tablet.cc:1722] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:04.651892 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=5da51dcf11ea4f6993ae5ef6fb739f92] alter complete (version 3)
I20240501 19:57:04.755501 24453 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:04.755646 24453 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:04.758360 24453 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet ff8157a5f1604dc9b9267fe330ebe1e3
I20240501 19:57:04.759670 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8157a5f1604dc9b9267fe330ebe1e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:04 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:04.759876 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8157a5f1604dc9b9267fe330ebe1e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:04 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:04.759939 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8157a5f1604dc9b9267fe330ebe1e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:04 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:04.767360 1711 tablet_replica.cc:330] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:04.767843 1713 tablet_replica.cc:330] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:04.768081 1713 raft_consensus.cc:2229] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:04.768175 1712 tablet_replica.cc:330] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:04.768325 1713 raft_consensus.cc:2258] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:04.769119 1713 ts_tablet_manager.cc:1875] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:04.769383 1711 raft_consensus.cc:2229] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:04.769539 1711 raft_consensus.cc:2258] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:04.769799 1712 raft_consensus.cc:2229] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:04.770097 1712 raft_consensus.cc:2258] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:04.770852 1711 ts_tablet_manager.cc:1875] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:04.773411 1711 ts_tablet_manager.cc:1888] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:04.773528 1711 log.cc:1196] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/ff8157a5f1604dc9b9267fe330ebe1e3
I20240501 19:57:04.773952 1711 ts_tablet_manager.cc:1909] T ff8157a5f1604dc9b9267fe330ebe1e3 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:57:04.775148 1712 ts_tablet_manager.cc:1875] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:04.775641 1713 ts_tablet_manager.cc:1888] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:04.775782 1713 log.cc:1196] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/ff8157a5f1604dc9b9267fe330ebe1e3
I20240501 19:57:04.776234 1713 ts_tablet_manager.cc:1909] T ff8157a5f1604dc9b9267fe330ebe1e3 P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:57:04.778926 1712 ts_tablet_manager.cc:1888] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:04.778919 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet ff8157a5f1604dc9b9267fe330ebe1e3 (table rebalancer_dynamic_table_new_name [id=5da51dcf11ea4f6993ae5ef6fb739f92]) successfully deleted
I20240501 19:57:04.779529 1712 log.cc:1196] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/ff8157a5f1604dc9b9267fe330ebe1e3
I20240501 19:57:04.779906 1712 ts_tablet_manager.cc:1909] T ff8157a5f1604dc9b9267fe330ebe1e3 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:57:04.781040 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet ff8157a5f1604dc9b9267fe330ebe1e3 (table rebalancer_dynamic_table_new_name [id=5da51dcf11ea4f6993ae5ef6fb739f92]) successfully deleted
I20240501 19:57:04.787798 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet ff8157a5f1604dc9b9267fe330ebe1e3 (table rebalancer_dynamic_table_new_name [id=5da51dcf11ea4f6993ae5ef6fb739f92]) successfully deleted
I20240501 19:57:04.860921 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:04.868510 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5aa0b8414d48868901834ec5469a3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2]), partition=
I20240501 19:57:04.868646 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5aa0b8414d48868901834ec5469a3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2]), partition=
I20240501 19:57:04.868783 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5aa0b8414d48868901834ec5469a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.868896 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5aa0b8414d48868901834ec5469a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.869599 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5aa0b8414d48868901834ec5469a3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2]), partition=
I20240501 19:57:04.869880 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5aa0b8414d48868901834ec5469a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.873524 1747 tablet_bootstrap.cc:492] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:04.878242 1747 tablet_bootstrap.cc:654] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.879729 1748 tablet_bootstrap.cc:492] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:04.880103 27267 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:04.880358 1746 tablet_bootstrap.cc:492] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:04.881362 1748 tablet_bootstrap.cc:654] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.885885 1746 tablet_bootstrap.cc:654] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.887256 1746 tablet_bootstrap.cc:492] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:04.887359 1746 ts_tablet_manager.cc:1367] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:04.887863 1746 raft_consensus.cc:348] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.888602 1748 tablet_bootstrap.cc:492] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:04.888698 1748 ts_tablet_manager.cc:1367] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:04.888949 27011 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:04.889173 1748 raft_consensus.cc:348] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.889313 1748 raft_consensus.cc:374] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.889346 1748 raft_consensus.cc:729] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.889467 1748 consensus_queue.cc:260] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.889756 1748 ts_tablet_manager.cc:1398] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.889921 1746 raft_consensus.cc:374] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.889973 1746 raft_consensus.cc:729] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.890081 1746 consensus_queue.cc:260] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.890390 1746 ts_tablet_manager.cc:1398] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:04.891445 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:04.892069 1747 tablet_bootstrap.cc:492] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:04.892180 1747 ts_tablet_manager.cc:1367] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240501 19:57:04.892578 1747 raft_consensus.cc:348] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.892772 1747 raft_consensus.cc:374] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.892822 1747 raft_consensus.cc:729] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.892976 1747 consensus_queue.cc:260] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.893220 1747 ts_tablet_manager.cc:1398] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:04.902915 24836 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:04.910223 1200 raft_consensus.cc:482] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:04.910318 1200 raft_consensus.cc:504] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.910710 1200 leader_election.cc:288] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:04.911083 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5aa0b8414d48868901834ec5469a3f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:04.911173 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5aa0b8414d48868901834ec5469a3f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:04.911206 24594 raft_consensus.cc:2454] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:04.911278 24727 raft_consensus.cc:2454] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:04.911684 27096 leader_election.cc:302] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:04.911916 1200 raft_consensus.cc:2781] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:04.911968 1200 raft_consensus.cc:482] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:04.912003 1200 raft_consensus.cc:3037] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.912911 1200 raft_consensus.cc:504] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.913221 1200 leader_election.cc:288] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:04.913462 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5aa0b8414d48868901834ec5469a3f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:04.913566 24594 raft_consensus.cc:3037] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.913707 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5aa0b8414d48868901834ec5469a3f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:04.913810 24727 raft_consensus.cc:3037] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.914515 24594 raft_consensus.cc:2454] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:04.914582 24727 raft_consensus.cc:2454] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:04.914810 27096 leader_election.cc:302] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:04.914817 1670 consensus_queue.cc:1035] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.915237 1200 raft_consensus.cc:2781] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:04.915382 1200 raft_consensus.cc:686] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:04.915535 1200 consensus_queue.cc:237] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:04.917963 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:04.917919 1671 consensus_queue.cc:1035] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.917912 24449 catalog_manager.cc:5484] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:04.935849 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:04.935952 24574 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:04.938419 1666 consensus_queue.cc:1035] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.950265 925 consensus_queue.cc:1035] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.954218 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:04.963255 1667 consensus_queue.cc:1035] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.975781 1666 consensus_queue.cc:1035] T 7a0bf1ddc7aa4a5ea4a7a863337e40d1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:04.985702 27031 raft_consensus.cc:1261] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:04.986321 731 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:05.003752 24594 consensus_queue.cc:237] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:05.005141 27163 raft_consensus.cc:1261] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:05.005311 24857 raft_consensus.cc:1261] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:05.005548 1760 consensus_queue.cc:1035] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:05.005753 1202 consensus_queue.cc:1035] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:05.007656 1202 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:05.009186 27163 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:05.009157 24857 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:05.012584 24453 catalog_manager.cc:5484] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 4 to 5, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:05.015091 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 0064e3e3ae1548bba511bd76fb4ec3e1 with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:05.015398 1789 tablet_replica.cc:330] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:05.015724 1789 raft_consensus.cc:2229] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:05.015874 1789 raft_consensus.cc:2258] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:05.016304 1789 ts_tablet_manager.cc:1875] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:05.018322 1789 ts_tablet_manager.cc:1888] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:05.018393 1789 log.cc:1196] T 0064e3e3ae1548bba511bd76fb4ec3e1 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/0064e3e3ae1548bba511bd76fb4ec3e1
I20240501 19:57:05.019047 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 0064e3e3ae1548bba511bd76fb4ec3e1 (table rebalance_test_table_13 [id=6557f4d959fd41d1adb6493b99703660]) successfully deleted
I20240501 19:57:05.050808 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:05.056073 24727 raft_consensus.cc:1261] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:05.056469 24594 raft_consensus.cc:1261] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:05.057121 1200 consensus_queue.cc:1035] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.057421 1200 consensus_queue.cc:1035] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.062079 24449 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 4b5aa0b8414d48868901834ec5469a3f (table rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2]). Expected version 1 got 0
I20240501 19:57:05.063230 24452 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 4b5aa0b8414d48868901834ec5469a3f (table rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2]). Expected version 1 got 0
I20240501 19:57:05.068431 1632 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:05.068768 1632 tablet.cc:1620] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:05.067605 1830 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:05.069167 1830 tablet.cc:1620] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:05.071087 24448 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 4b5aa0b8414d48868901834ec5469a3f (table rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2]). Expected version 1 got 0
I20240501 19:57:05.071806 24453 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 4b5aa0b8414d48868901834ec5469a3f (table rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2]). Expected version 1 got 0
I20240501 19:57:05.074971 1831 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:05.075428 1831 tablet.cc:1620] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:05.076611 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2] alter complete (version 1)
I20240501 19:57:05.089820 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:05.099454 24574 tablet_service.cc:1467] Processing CreateTablet for tablet c7b14aec624f480189365d112f838635 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.099774 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b14aec624f480189365d112f838635. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.100395 24966 tablet_service.cc:1467] Processing CreateTablet for tablet c7b14aec624f480189365d112f838635 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.100636 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b14aec624f480189365d112f838635. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.102088 1747 tablet_bootstrap.cc:492] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:05.103122 1747 tablet_bootstrap.cc:654] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.103350 1832 tablet_bootstrap.cc:492] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:05.104218 27405 tablet_service.cc:1467] Processing CreateTablet for tablet c7b14aec624f480189365d112f838635 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.104496 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b14aec624f480189365d112f838635. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.104633 27143 tablet_service.cc:1467] Processing CreateTablet for tablet a4ade4acdb974d5586bf8bc8a6589f1c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.104880 1832 tablet_bootstrap.cc:654] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.104916 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ade4acdb974d5586bf8bc8a6589f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.106128 1832 tablet_bootstrap.cc:492] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:05.106199 1832 ts_tablet_manager.cc:1367] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:05.106720 1746 tablet_bootstrap.cc:492] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:05.106654 1832 raft_consensus.cc:348] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.106782 1832 raft_consensus.cc:374] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.106822 1832 raft_consensus.cc:729] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.106935 1832 consensus_queue.cc:260] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.107247 1832 ts_tablet_manager.cc:1398] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:05.107645 1746 tablet_bootstrap.cc:654] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.108407 1835 tablet_bootstrap.cc:492] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:05.109618 1835 tablet_bootstrap.cc:654] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.110853 1746 tablet_bootstrap.cc:492] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:05.110931 1746 ts_tablet_manager.cc:1367] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:05.111371 1746 raft_consensus.cc:348] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.111471 1746 raft_consensus.cc:374] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.111502 1746 raft_consensus.cc:729] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.111604 1746 consensus_queue.cc:260] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.111893 1746 ts_tablet_manager.cc:1398] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:05.112336 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 38626dc7814f454e85f243fed9607221 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.112500 27267 tablet_service.cc:1467] Processing CreateTablet for tablet a4ade4acdb974d5586bf8bc8a6589f1c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.112589 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38626dc7814f454e85f243fed9607221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.112764 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ade4acdb974d5586bf8bc8a6589f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.113118 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 38626dc7814f454e85f243fed9607221 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.113313 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38626dc7814f454e85f243fed9607221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.114667 1838 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:05.115868 1838 tablet_bootstrap.cc:654] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.115934 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 38626dc7814f454e85f243fed9607221 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.116147 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38626dc7814f454e85f243fed9607221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.118233 1832 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:05.119148 1832 tablet_bootstrap.cc:654] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.119148 1746 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:05.120216 1746 tablet_bootstrap.cc:654] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.121111 1747 tablet_bootstrap.cc:492] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:05.121209 1832 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:05.121233 1747 ts_tablet_manager.cc:1367] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240501 19:57:05.121289 1832 ts_tablet_manager.cc:1367] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:05.121716 1832 raft_consensus.cc:348] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.121819 1832 raft_consensus.cc:374] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.121876 1832 raft_consensus.cc:729] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.121999 1832 consensus_queue.cc:260] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.122068 1747 raft_consensus.cc:348] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.122171 1747 raft_consensus.cc:374] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.122205 1747 raft_consensus.cc:729] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.122334 1832 ts_tablet_manager.cc:1398] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:05.122318 1747 consensus_queue.cc:260] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.122601 1747 ts_tablet_manager.cc:1398] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:05.122982 24574 tablet_service.cc:1467] Processing CreateTablet for tablet a4ade4acdb974d5586bf8bc8a6589f1c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.123869 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ade4acdb974d5586bf8bc8a6589f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.124575 1746 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:05.124694 1746 ts_tablet_manager.cc:1367] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:05.125283 1746 raft_consensus.cc:348] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.125555 1746 raft_consensus.cc:374] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.125634 1746 raft_consensus.cc:729] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.125797 1746 consensus_queue.cc:260] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.126891 1746 ts_tablet_manager.cc:1398] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:05.127925 1835 tablet_bootstrap.cc:492] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:05.128063 1835 ts_tablet_manager.cc:1367] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:57:05.129804 1747 tablet_bootstrap.cc:492] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:05.131089 24857 raft_consensus.cc:1261] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:05.131686 1747 tablet_bootstrap.cc:654] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.131673 1562 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:05.133082 1839 tablet_bootstrap.cc:492] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:05.133823 1835 raft_consensus.cc:348] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.133991 1835 raft_consensus.cc:374] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.134068 1835 raft_consensus.cc:729] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.134253 1835 consensus_queue.cc:260] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.135947 1839 tablet_bootstrap.cc:654] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.136544 1747 tablet_bootstrap.cc:492] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:05.136626 1747 ts_tablet_manager.cc:1367] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:57:05.135481 1835 ts_tablet_manager.cc:1398] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:05.137084 1747 raft_consensus.cc:348] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.137205 1747 raft_consensus.cc:374] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.137244 1747 raft_consensus.cc:729] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.137364 1747 consensus_queue.cc:260] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.137679 1747 ts_tablet_manager.cc:1398] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:05.138170 1838 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:05.138250 1838 ts_tablet_manager.cc:1367] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 19:57:05.138707 1838 raft_consensus.cc:348] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.140508 24727 raft_consensus.cc:1261] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:05.141552 731 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:05.142108 1838 raft_consensus.cc:374] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.142171 1838 raft_consensus.cc:729] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.142354 1838 consensus_queue.cc:260] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.142773 1838 ts_tablet_manager.cc:1398] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:05.147421 1839 tablet_bootstrap.cc:492] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:05.147516 1839 ts_tablet_manager.cc:1367] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:57:05.148108 1839 raft_consensus.cc:348] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.148228 1839 raft_consensus.cc:374] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.148254 1839 raft_consensus.cc:729] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.148545 1839 consensus_queue.cc:260] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.149001 1839 ts_tablet_manager.cc:1398] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:05.171835 27011 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:05.176115 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:05.183218 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:05.190923 27267 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:05.191435 1830 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:05.191735 1830 tablet.cc:1620] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:05.191989 1831 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:05.192229 1831 tablet.cc:1620] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:05.193660 1632 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:05.193954 1632 tablet.cc:1620] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:05.200874 24448 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=22e82747f22f42128ef9afe2083019d2] alter complete (version 2)
I20240501 19:57:05.203195 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:05.215831 24836 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:05.216606 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:05.228427 1202 raft_consensus.cc:482] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:05.228552 1202 raft_consensus.cc:504] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.228931 1202 leader_election.cc:288] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:05.229418 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ade4acdb974d5586bf8bc8a6589f1c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:05.229568 27276 raft_consensus.cc:2454] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:05.230067 24531 leader_election.cc:302] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:05.230067 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ade4acdb974d5586bf8bc8a6589f1c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:05.230207 27163 raft_consensus.cc:2454] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:05.230453 1760 raft_consensus.cc:2781] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:05.230513 1760 raft_consensus.cc:482] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:05.230551 1760 raft_consensus.cc:3037] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.231006 24574 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:05.231757 1760 raft_consensus.cc:504] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.231825 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:05.232077 1760 leader_election.cc:288] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:05.232813 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ade4acdb974d5586bf8bc8a6589f1c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:05.232919 27163 raft_consensus.cc:3037] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.233899 27163 raft_consensus.cc:2454] T a4ade4acdb974d5586bf8bc8a6589f1c P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:05.236516 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ade4acdb974d5586bf8bc8a6589f1c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:05.236647 27278 raft_consensus.cc:3037] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.237665 27278 raft_consensus.cc:2454] T a4ade4acdb974d5586bf8bc8a6589f1c P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:05.238685 24531 leader_election.cc:302] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:05.238992 1202 raft_consensus.cc:2781] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:05.239089 1202 raft_consensus.cc:686] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:05.239214 1202 consensus_queue.cc:237] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.251953 24453 catalog_manager.cc:5484] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:05.295809 27031 consensus_queue.cc:237] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.297006 24595 raft_consensus.cc:1261] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:05.297060 27278 raft_consensus.cc:1261] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:05.297387 1665 consensus_queue.cc:1035] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:05.297578 1666 consensus_queue.cc:1035] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:05.299471 1665 raft_consensus.cc:2932] T 8eb1e61bb542415caac5978943e6b3eb P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:05.299823 24594 raft_consensus.cc:2932] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:05.299896 27278 raft_consensus.cc:2932] T 8eb1e61bb542415caac5978943e6b3eb P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:05.302902 24453 catalog_manager.cc:5484] T 8eb1e61bb542415caac5978943e6b3eb P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:05.305167 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:05.306386 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb1e61bb542415caac5978943e6b3eb with delete_type TABLET_DATA_TOMBSTONED (TS 4a6e54e86a9949e5970b42e8e32e7561 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:05.309403 1885 tablet_replica.cc:330] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:05.309597 1885 raft_consensus.cc:2229] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:05.309775 1885 raft_consensus.cc:2258] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:05.310266 1885 ts_tablet_manager.cc:1875] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:05.312798 1830 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:05.313386 1885 ts_tablet_manager.cc:1888] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:05.313500 1885 log.cc:1196] T 8eb1e61bb542415caac5978943e6b3eb P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/8eb1e61bb542415caac5978943e6b3eb
I20240501 19:57:05.315135 1632 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:05.317924 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 8eb1e61bb542415caac5978943e6b3eb (table rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]) successfully deleted
I20240501 19:57:05.319067 1831 tablet.cc:1722] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:05.318887 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=22e82747f22f42128ef9afe2083019d2] alter complete (version 3)
I20240501 19:57:05.414172 1760 raft_consensus.cc:482] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:05.414289 1760 raft_consensus.cc:504] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.414733 1760 leader_election.cc:288] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:05.415119 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b14aec624f480189365d112f838635" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:05.415262 27425 raft_consensus.cc:2454] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:05.415547 24531 leader_election.cc:302] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:05.416010 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b14aec624f480189365d112f838635" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:05.416136 24987 raft_consensus.cc:2454] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:05.416505 1760 raft_consensus.cc:2781] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:05.416628 1760 raft_consensus.cc:482] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:05.416735 1760 raft_consensus.cc:3037] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.423050 24452 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:05.423188 24452 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:05.423539 1760 raft_consensus.cc:504] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.424115 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b14aec624f480189365d112f838635" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:05.424228 27425 raft_consensus.cc:3037] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.425259 27425 raft_consensus.cc:2454] T c7b14aec624f480189365d112f838635 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:05.426144 24531 leader_election.cc:302] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:05.426321 1202 raft_consensus.cc:2781] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:05.426432 1202 raft_consensus.cc:686] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:05.426566 24452 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 4b5aa0b8414d48868901834ec5469a3f
I20240501 19:57:05.426580 1760 leader_election.cc:288] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:05.426596 1202 consensus_queue.cc:237] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:05.427645 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b14aec624f480189365d112f838635" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:05.427778 24987 raft_consensus.cc:3037] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.428274 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5aa0b8414d48868901834ec5469a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:05 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:05.428879 24987 raft_consensus.cc:2454] T c7b14aec624f480189365d112f838635 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:05.429502 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5aa0b8414d48868901834ec5469a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:05 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:05.430904 1934 tablet_replica.cc:330] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:05.431389 1934 raft_consensus.cc:2229] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:05.431576 1934 raft_consensus.cc:2258] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:05.431890 24454 catalog_manager.cc:5484] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:05.432914 1934 ts_tablet_manager.cc:1875] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:05.434070 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5aa0b8414d48868901834ec5469a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:05 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:05.435588 1934 ts_tablet_manager.cc:1888] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:05.435675 1934 log.cc:1196] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b5aa0b8414d48868901834ec5469a3f
I20240501 19:57:05.436046 1934 ts_tablet_manager.cc:1909] T 4b5aa0b8414d48868901834ec5469a3f P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:57:05.436717 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 4b5aa0b8414d48868901834ec5469a3f (table rebalancer_dynamic_table_new_name [id=22e82747f22f42128ef9afe2083019d2]) successfully deleted
I20240501 19:57:05.436745 1935 tablet_replica.cc:330] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:05.436903 1935 raft_consensus.cc:2229] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:05.437120 1935 raft_consensus.cc:2258] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:05.437695 1935 ts_tablet_manager.cc:1875] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:05.439481 1933 tablet_replica.cc:330] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:05.439671 1933 raft_consensus.cc:2229] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:05.439839 1933 raft_consensus.cc:2258] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:05.440447 1933 ts_tablet_manager.cc:1875] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:05.442090 1935 ts_tablet_manager.cc:1888] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:05.442157 1935 log.cc:1196] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/4b5aa0b8414d48868901834ec5469a3f
I20240501 19:57:05.442441 1935 ts_tablet_manager.cc:1909] T 4b5aa0b8414d48868901834ec5469a3f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:57:05.443478 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 4b5aa0b8414d48868901834ec5469a3f (table rebalancer_dynamic_table_new_name [id=22e82747f22f42128ef9afe2083019d2]) successfully deleted
I20240501 19:57:05.443884 1933 ts_tablet_manager.cc:1888] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:05.443994 1933 log.cc:1196] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/4b5aa0b8414d48868901834ec5469a3f
I20240501 19:57:05.444324 1933 ts_tablet_manager.cc:1909] T 4b5aa0b8414d48868901834ec5469a3f P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:57:05.444955 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 4b5aa0b8414d48868901834ec5469a3f (table rebalancer_dynamic_table_new_name [id=22e82747f22f42128ef9afe2083019d2]) successfully deleted
I20240501 19:57:05.448474 1877 raft_consensus.cc:482] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:05.448563 1877 raft_consensus.cc:504] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.448976 1877 leader_election.cc:288] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:05.449335 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38626dc7814f454e85f243fed9607221" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:05.449380 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38626dc7814f454e85f243fed9607221" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:05.449489 27163 raft_consensus.cc:2454] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:05.449517 24987 raft_consensus.cc:2454] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:05.449792 26966 leader_election.cc:302] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:05.449931 1877 raft_consensus.cc:2781] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:05.449990 1877 raft_consensus.cc:482] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:05.450038 1877 raft_consensus.cc:3037] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.451153 1877 raft_consensus.cc:504] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.451485 1877 leader_election.cc:288] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:05.452757 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38626dc7814f454e85f243fed9607221" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:05.452867 27163 raft_consensus.cc:3037] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.452904 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38626dc7814f454e85f243fed9607221" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:05.453009 24987 raft_consensus.cc:3037] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.453812 27163 raft_consensus.cc:2454] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:05.453967 24987 raft_consensus.cc:2454] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:05.454557 26966 leader_election.cc:302] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:05.454711 1877 raft_consensus.cc:2781] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:05.454793 1877 raft_consensus.cc:686] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:05.454905 1877 consensus_queue.cc:237] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:05.457255 24454 catalog_manager.cc:5484] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:05.477622 24836 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:05.480412 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:05.481525 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:05.483548 27011 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:05.486477 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:05.498875 24574 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:05.498966 27267 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:05.500782 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:05.535120 24452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:05.540791 24836 tablet_service.cc:1467] Processing CreateTablet for tablet b8706ea9ff0d4ea48148573d0efb4c3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9146f6230f464020b6e264be9fd8e302]), partition=
I20240501 19:57:05.540776 24705 tablet_service.cc:1467] Processing CreateTablet for tablet b8706ea9ff0d4ea48148573d0efb4c3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9146f6230f464020b6e264be9fd8e302]), partition=
I20240501 19:57:05.541035 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8706ea9ff0d4ea48148573d0efb4c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.541035 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8706ea9ff0d4ea48148573d0efb4c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.543325 27011 tablet_service.cc:1467] Processing CreateTablet for tablet b8706ea9ff0d4ea48148573d0efb4c3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9146f6230f464020b6e264be9fd8e302]), partition=
I20240501 19:57:05.543573 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8706ea9ff0d4ea48148573d0efb4c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.543723 1948 tablet_bootstrap.cc:492] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:05.544700 1948 tablet_bootstrap.cc:654] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.545850 1948 tablet_bootstrap.cc:492] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:05.545913 1838 tablet_bootstrap.cc:492] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:05.545925 1948 ts_tablet_manager.cc:1367] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:05.546339 1948 raft_consensus.cc:348] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.546437 1948 raft_consensus.cc:374] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.546468 1948 raft_consensus.cc:729] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.546619 1948 consensus_queue.cc:260] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.546769 1838 tablet_bootstrap.cc:654] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.546975 1948 ts_tablet_manager.cc:1398] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:05.547945 1838 tablet_bootstrap.cc:492] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:05.548044 1838 ts_tablet_manager.cc:1367] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:57:05.548383 1838 raft_consensus.cc:348] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.548458 1838 raft_consensus.cc:374] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.548477 1838 raft_consensus.cc:729] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.548563 1838 consensus_queue.cc:260] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.548790 1838 ts_tablet_manager.cc:1398] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:05.549567 1951 tablet_bootstrap.cc:492] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:05.550348 1951 tablet_bootstrap.cc:654] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.555897 1951 tablet_bootstrap.cc:492] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:05.555985 1951 ts_tablet_manager.cc:1367] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:05.556406 1951 raft_consensus.cc:348] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.556509 1951 raft_consensus.cc:374] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.556545 1951 raft_consensus.cc:729] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.556694 1951 consensus_queue.cc:260] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.556969 1951 ts_tablet_manager.cc:1398] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:05.566900 1859 raft_consensus.cc:482] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:05.566987 1859 raft_consensus.cc:504] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.567507 1859 leader_election.cc:288] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:05.568214 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8706ea9ff0d4ea48148573d0efb4c3e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:05.568244 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8706ea9ff0d4ea48148573d0efb4c3e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:05.568341 27031 raft_consensus.cc:2454] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:05.568344 24856 raft_consensus.cc:2454] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:05.568604 24660 leader_election.cc:302] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:05.569041 1859 raft_consensus.cc:2781] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:05.569101 1859 raft_consensus.cc:482] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:05.569161 1859 raft_consensus.cc:3037] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.570148 1859 raft_consensus.cc:504] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.570525 1859 leader_election.cc:288] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:05.571240 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8706ea9ff0d4ea48148573d0efb4c3e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:05.571339 27031 raft_consensus.cc:3037] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.571414 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8706ea9ff0d4ea48148573d0efb4c3e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:05.571535 24856 raft_consensus.cc:3037] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.572211 27031 raft_consensus.cc:2454] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:05.572414 24856 raft_consensus.cc:2454] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:05.572460 24660 leader_election.cc:302] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:05.572599 1859 raft_consensus.cc:2781] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:05.572690 1859 raft_consensus.cc:686] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:05.572803 1859 consensus_queue.cc:237] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:05.574826 24454 catalog_manager.cc:5484] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:05.599341 24594 consensus_queue.cc:237] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:05.600519 27031 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:05.600584 24856 raft_consensus.cc:1261] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:05.600878 1536 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:05.601042 1202 consensus_queue.cc:1035] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:05.604372 1536 raft_consensus.cc:2932] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:05.604737 24856 raft_consensus.cc:2932] T 224ae792734e4d1ab3c28e20d82fdf14 P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:05.607414 27031 raft_consensus.cc:2932] T 224ae792734e4d1ab3c28e20d82fdf14 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:05.607666 24452 catalog_manager.cc:5484] T 224ae792734e4d1ab3c28e20d82fdf14 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:05.610689 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 224ae792734e4d1ab3c28e20d82fdf14 with delete_type TABLET_DATA_TOMBSTONED (TS 9081292597be4a72a42b216fddcd7971 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:05.611265 1934 tablet_replica.cc:330] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:05.611407 1934 raft_consensus.cc:2229] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:05.611521 1934 raft_consensus.cc:2258] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:05.612084 1934 ts_tablet_manager.cc:1875] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:05.613997 1934 ts_tablet_manager.cc:1888] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:57:05.614045 1934 log.cc:1196] T 224ae792734e4d1ab3c28e20d82fdf14 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/224ae792734e4d1ab3c28e20d82fdf14
I20240501 19:57:05.614485 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 224ae792734e4d1ab3c28e20d82fdf14 (table rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]) successfully deleted
I20240501 19:57:05.620411 1536 consensus_queue.cc:1035] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.624783 1760 consensus_queue.cc:1035] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.641607 24725 consensus_queue.cc:237] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:05.642947 24856 raft_consensus.cc:1261] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:05.643229 24662 consensus_peers.cc:485] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1cd92f84a944b94b5948123e3e1d7a1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:05.643370 1290 consensus_queue.cc:1035] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:05.643884 24987 raft_consensus.cc:1261] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:05.644258 1290 consensus_queue.cc:1035] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:05.645713 1859 raft_consensus.cc:2932] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:05.646097 24987 raft_consensus.cc:2932] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:05.647500 24856 raft_consensus.cc:2932] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:05.649170 24454 catalog_manager.cc:5484] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:05.675163 27031 consensus_queue.cc:237] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:05.676342 24595 raft_consensus.cc:1261] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:05.676642 24856 raft_consensus.cc:1261] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:05.676837 1665 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:05.676990 26967 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d29130c48f7a435498e854973a090b37. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:05.677135 1877 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:05.678941 1665 raft_consensus.cc:2932] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:05.679302 24856 raft_consensus.cc:2932] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:05.682564 24595 raft_consensus.cc:2932] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:05.682751 24447 catalog_manager.cc:5484] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:05.688856 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:05.692888 24856 raft_consensus.cc:1261] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:05.692898 27031 raft_consensus.cc:1261] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:05.693346 1290 consensus_queue.cc:1035] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.693542 32475 consensus_queue.cc:1035] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.696228 1831 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:05.696565 1831 tablet.cc:1620] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:05.696645 24452 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet b8706ea9ff0d4ea48148573d0efb4c3e (table rebalancer_dynamic_table [id=9146f6230f464020b6e264be9fd8e302]). Expected version 1 got 0
I20240501 19:57:05.698341 24447 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet b8706ea9ff0d4ea48148573d0efb4c3e (table rebalancer_dynamic_table [id=9146f6230f464020b6e264be9fd8e302]). Expected version 1 got 0
I20240501 19:57:05.698551 2034 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:05.698984 2034 tablet.cc:1620] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:05.703115 2031 mvcc.cc:205] Tried to move back new op lower bound from 7022974671633502208 to 7022974671147290624. Current Snapshot: MvccSnapshot[applied={T|T < 7022974671633502208}]
I20240501 19:57:05.705067 24454 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet b8706ea9ff0d4ea48148573d0efb4c3e (table rebalancer_dynamic_table [id=9146f6230f464020b6e264be9fd8e302]). Expected version 1 got 0
I20240501 19:57:05.705431 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=9146f6230f464020b6e264be9fd8e302] alter complete (version 1)
I20240501 19:57:05.705796 2035 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:05.706466 2035 tablet.cc:1620] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:05.806749 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:05.818302 1831 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:05.818652 1831 tablet.cc:1620] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:05.820282 2034 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:05.820585 2034 tablet.cc:1620] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:05.824731 24574 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:05.825879 2035 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:05.826191 2035 tablet.cc:1620] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:05.826944 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=9146f6230f464020b6e264be9fd8e302] alter complete (version 2)
I20240501 19:57:05.833734 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:05.834057 27011 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:05.837163 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:05.838654 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:05.839994 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:05.847155 27267 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:05.858105 2018 consensus_queue.cc:1035] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.863123 1760 consensus_queue.cc:1035] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.872258 1877 consensus_queue.cc:1035] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.882310 1760 consensus_queue.cc:1035] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:05.891202 24836 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:05.934002 24446 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:05.939410 1831 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:05.939747 2034 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:05.940297 2035 tablet.cc:1722] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:05.945070 24446 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=9146f6230f464020b6e264be9fd8e302] alter complete (version 3)
I20240501 19:57:06.009091 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:06.020545 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 3b46d4892fda4ae3b749460bdbb2050e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.020866 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b46d4892fda4ae3b749460bdbb2050e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.021029 27142 tablet_service.cc:1467] Processing CreateTablet for tablet e19523b5482d4e5a82252f77779afb6d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.021034 24836 tablet_service.cc:1467] Processing CreateTablet for tablet ac83705168fc418c9672406a78981279 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.021248 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19523b5482d4e5a82252f77779afb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.021261 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac83705168fc418c9672406a78981279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.022961 1948 tablet_bootstrap.cc:492] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:06.023447 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 3b46d4892fda4ae3b749460bdbb2050e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.023486 27404 tablet_service.cc:1467] Processing CreateTablet for tablet e19523b5482d4e5a82252f77779afb6d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.023726 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b46d4892fda4ae3b749460bdbb2050e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.023746 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19523b5482d4e5a82252f77779afb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.023977 1948 tablet_bootstrap.cc:654] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.024744 27405 tablet_service.cc:1467] Processing CreateTablet for tablet ac83705168fc418c9672406a78981279 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.024917 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac83705168fc418c9672406a78981279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.025981 2122 tablet_bootstrap.cc:492] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:06.026399 27267 tablet_service.cc:1467] Processing CreateTablet for tablet ac83705168fc418c9672406a78981279 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.026700 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac83705168fc418c9672406a78981279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.026922 2121 tablet_bootstrap.cc:492] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:06.027519 1948 tablet_bootstrap.cc:492] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:06.027593 2122 tablet_bootstrap.cc:654] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.027606 1948 ts_tablet_manager.cc:1367] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:06.028131 1948 raft_consensus.cc:348] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.028229 1948 raft_consensus.cc:374] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.028260 1948 raft_consensus.cc:729] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.028394 1948 consensus_queue.cc:260] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.029284 2132 tablet_bootstrap.cc:492] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:06.029538 2121 tablet_bootstrap.cc:654] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.030841 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 3b46d4892fda4ae3b749460bdbb2050e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.031750 2122 tablet_bootstrap.cc:492] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:06.031841 2122 ts_tablet_manager.cc:1367] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:06.032389 2122 raft_consensus.cc:348] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.032509 2122 raft_consensus.cc:374] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.032545 2122 raft_consensus.cc:729] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.032558 27267 tablet_service.cc:1467] Processing CreateTablet for tablet e19523b5482d4e5a82252f77779afb6d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=78e9351ecbe44a29b5f3c7e022830d45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.032722 2122 consensus_queue.cc:260] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.032819 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19523b5482d4e5a82252f77779afb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.033026 2132 tablet_bootstrap.cc:654] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.033248 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b46d4892fda4ae3b749460bdbb2050e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.034210 2121 tablet_bootstrap.cc:492] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:06.034222 2132 tablet_bootstrap.cc:492] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:06.034289 2121 ts_tablet_manager.cc:1367] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:06.034293 2132 ts_tablet_manager.cc:1367] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:06.034718 2122 ts_tablet_manager.cc:1398] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:06.034749 2132 raft_consensus.cc:348] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.034849 2132 raft_consensus.cc:374] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.035025 2132 raft_consensus.cc:729] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.035187 2121 raft_consensus.cc:348] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.035243 2132 consensus_queue.cc:260] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.035293 2121 raft_consensus.cc:374] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.035331 2121 raft_consensus.cc:729] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.035717 2121 consensus_queue.cc:260] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.036123 2121 ts_tablet_manager.cc:1398] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:06.036314 2121 tablet_bootstrap.cc:492] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:06.037341 2121 tablet_bootstrap.cc:654] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.037884 2132 ts_tablet_manager.cc:1398] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:06.038097 2132 tablet_bootstrap.cc:492] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:06.040705 2132 tablet_bootstrap.cc:654] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.043097 2132 tablet_bootstrap.cc:492] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:06.043177 2132 ts_tablet_manager.cc:1367] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:06.043615 2132 raft_consensus.cc:348] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.043709 2132 raft_consensus.cc:374] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.043735 2132 raft_consensus.cc:729] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.043838 2132 consensus_queue.cc:260] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.044085 2132 ts_tablet_manager.cc:1398] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:06.045468 1951 tablet_bootstrap.cc:492] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:06.046744 1951 tablet_bootstrap.cc:654] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.048718 24452 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:06.048846 24452 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:06.051393 1951 tablet_bootstrap.cc:492] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:06.051489 1951 ts_tablet_manager.cc:1367] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:06.051952 1951 raft_consensus.cc:348] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.052047 1951 raft_consensus.cc:374] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.052073 1951 raft_consensus.cc:729] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.052175 1951 consensus_queue.cc:260] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.052434 1951 ts_tablet_manager.cc:1398] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:06.053236 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:06.054433 24452 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet b8706ea9ff0d4ea48148573d0efb4c3e
I20240501 19:57:06.054939 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet b8706ea9ff0d4ea48148573d0efb4c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:06 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:06.056265 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet b8706ea9ff0d4ea48148573d0efb4c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:06 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:06.057050 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet b8706ea9ff0d4ea48148573d0efb4c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:06 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:06.057246 1934 tablet_replica.cc:330] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:06.057397 1934 raft_consensus.cc:2229] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:06.057581 1934 raft_consensus.cc:2258] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:06.058125 1934 ts_tablet_manager.cc:1875] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:06.060161 1934 ts_tablet_manager.cc:1888] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:06.060231 1934 log.cc:1196] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8706ea9ff0d4ea48148573d0efb4c3e
I20240501 19:57:06.060441 1948 ts_tablet_manager.cc:1398] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.033s user 0.001s sys 0.000s
I20240501 19:57:06.060489 1934 ts_tablet_manager.cc:1909] T b8706ea9ff0d4ea48148573d0efb4c3e P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:57:06.061111 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet b8706ea9ff0d4ea48148573d0efb4c3e (table rebalancer_dynamic_table_new_name [id=9146f6230f464020b6e264be9fd8e302]) successfully deleted
I20240501 19:57:06.062440 24836 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:06.063323 27405 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:06.066949 2134 tablet_bootstrap.cc:492] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:06.067270 2142 tablet_replica.cc:330] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:06.067461 2142 raft_consensus.cc:2229] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:06.067590 2142 raft_consensus.cc:2258] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:06.068609 2134 tablet_bootstrap.cc:654] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.069031 2142 ts_tablet_manager.cc:1875] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:06.071306 2142 ts_tablet_manager.cc:1888] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:06.072549 2121 tablet_bootstrap.cc:492] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:06.072688 2121 ts_tablet_manager.cc:1367] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240501 19:57:06.072705 24574 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:06.073237 2121 raft_consensus.cc:348] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.073427 2121 raft_consensus.cc:374] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.073515 2121 raft_consensus.cc:729] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.073676 2121 consensus_queue.cc:260] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.074083 2121 ts_tablet_manager.cc:1398] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:06.074226 2142 log.cc:1196] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/b8706ea9ff0d4ea48148573d0efb4c3e
I20240501 19:57:06.074544 2142 ts_tablet_manager.cc:1909] T b8706ea9ff0d4ea48148573d0efb4c3e P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:06.075260 2143 tablet_replica.cc:330] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:06.075444 2143 raft_consensus.cc:2229] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:06.075572 2143 raft_consensus.cc:2258] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:06.076115 2143 ts_tablet_manager.cc:1875] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:06.078457 2143 ts_tablet_manager.cc:1888] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:06.078687 2143 log.cc:1196] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8706ea9ff0d4ea48148573d0efb4c3e
I20240501 19:57:06.079089 2143 ts_tablet_manager.cc:1909] T b8706ea9ff0d4ea48148573d0efb4c3e P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:57:06.080751 2134 tablet_bootstrap.cc:492] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:06.080899 2134 ts_tablet_manager.cc:1367] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:57:06.081378 2134 raft_consensus.cc:348] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.081492 2134 raft_consensus.cc:374] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.081537 2134 raft_consensus.cc:729] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.081668 2134 consensus_queue.cc:260] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.081928 2134 ts_tablet_manager.cc:1398] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:06.082509 2134 tablet_bootstrap.cc:492] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:06.083433 2134 tablet_bootstrap.cc:654] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.083675 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet b8706ea9ff0d4ea48148573d0efb4c3e (table rebalancer_dynamic_table_new_name [id=9146f6230f464020b6e264be9fd8e302]) successfully deleted
I20240501 19:57:06.084059 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet b8706ea9ff0d4ea48148573d0efb4c3e (table rebalancer_dynamic_table_new_name [id=9146f6230f464020b6e264be9fd8e302]) successfully deleted
I20240501 19:57:06.085556 27267 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:06.086757 27011 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:06.089026 24705 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:06.093616 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:06.099179 1859 raft_consensus.cc:482] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:06.099279 1859 raft_consensus.cc:504] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.099735 1859 leader_election.cc:288] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:06.103143 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b46d4892fda4ae3b749460bdbb2050e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:06.103292 24595 raft_consensus.cc:2454] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:06.103564 24660 leader_election.cc:302] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:06.103734 1859 raft_consensus.cc:2781] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:06.103775 1859 raft_consensus.cc:482] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:06.103825 1859 raft_consensus.cc:3037] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.104756 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b46d4892fda4ae3b749460bdbb2050e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:06.104837 1859 raft_consensus.cc:504] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.104882 27163 raft_consensus.cc:2454] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:06.105170 1859 leader_election.cc:288] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:06.105859 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b46d4892fda4ae3b749460bdbb2050e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:06.105949 27163 raft_consensus.cc:3037] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.106031 925 raft_consensus.cc:482] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:06.106123 925 raft_consensus.cc:504] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.106570 925 leader_election.cc:288] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:06.106899 27163 raft_consensus.cc:2454] T 3b46d4892fda4ae3b749460bdbb2050e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:06.107102 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac83705168fc418c9672406a78981279" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:06.107146 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac83705168fc418c9672406a78981279" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:06.107229 27276 raft_consensus.cc:2454] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:06.107272 27424 raft_consensus.cc:2454] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:06.107590 24793 leader_election.cc:302] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:06.107872 925 raft_consensus.cc:2781] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:06.107966 925 raft_consensus.cc:482] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:06.108048 925 raft_consensus.cc:3037] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.108354 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b46d4892fda4ae3b749460bdbb2050e" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:06.108460 24595 raft_consensus.cc:3037] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.108897 925 raft_consensus.cc:504] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.109218 925 leader_election.cc:288] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:06.109345 24595 raft_consensus.cc:2454] T 3b46d4892fda4ae3b749460bdbb2050e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:06.109652 24660 leader_election.cc:302] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:06.109709 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac83705168fc418c9672406a78981279" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:06.109732 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac83705168fc418c9672406a78981279" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:06.109819 1859 raft_consensus.cc:2781] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:06.109825 27424 raft_consensus.cc:3037] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.109862 27276 raft_consensus.cc:3037] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.110164 1859 raft_consensus.cc:686] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:06.110412 1859 consensus_queue.cc:237] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:06.110788 27424 raft_consensus.cc:2454] T ac83705168fc418c9672406a78981279 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:06.110879 27276 raft_consensus.cc:2454] T ac83705168fc418c9672406a78981279 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:06.113715 24793 leader_election.cc:302] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:06.114117 925 raft_consensus.cc:2781] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:06.114202 925 raft_consensus.cc:686] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:06.114346 925 consensus_queue.cc:237] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.116650 24451 catalog_manager.cc:5484] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:06.116928 24449 catalog_manager.cc:5484] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:06.139467 2134 tablet_bootstrap.cc:492] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:06.139577 2134 ts_tablet_manager.cc:1367] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.001s
I20240501 19:57:06.140041 2134 raft_consensus.cc:348] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.140156 2134 raft_consensus.cc:374] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.140187 2134 raft_consensus.cc:729] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.140307 2134 consensus_queue.cc:260] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.140576 2134 ts_tablet_manager.cc:1398] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:06.151633 1023 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:06.151736 1023 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.152086 1023 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:06.152750 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19523b5482d4e5a82252f77779afb6d" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:06.152884 27163 raft_consensus.cc:2454] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:06.153174 27361 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:06.153260 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19523b5482d4e5a82252f77779afb6d" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:06.153339 1023 raft_consensus.cc:2781] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:06.153386 1023 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:06.153393 27276 raft_consensus.cc:2454] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:06.153427 1023 raft_consensus.cc:3037] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.154548 1023 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.154884 1023 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:06.155390 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19523b5482d4e5a82252f77779afb6d" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:06.155504 27276 raft_consensus.cc:3037] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.155553 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19523b5482d4e5a82252f77779afb6d" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:06.155660 27163 raft_consensus.cc:3037] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.156512 27276 raft_consensus.cc:2454] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:06.156644 27163 raft_consensus.cc:2454] T e19523b5482d4e5a82252f77779afb6d P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:06.156924 27361 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:06.157102 2014 raft_consensus.cc:2781] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:06.157193 2014 raft_consensus.cc:686] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:57:06.157331 2014 consensus_queue.cc:237] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.158670 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:06.159529 24449 catalog_manager.cc:5484] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:06.165850 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 032eb50f77304faa8e86411eae7a11a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]), partition=
I20240501 19:57:06.165959 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 032eb50f77304faa8e86411eae7a11a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]), partition=
I20240501 19:57:06.165968 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 032eb50f77304faa8e86411eae7a11a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]), partition=
I20240501 19:57:06.166081 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032eb50f77304faa8e86411eae7a11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.166180 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032eb50f77304faa8e86411eae7a11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.166180 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032eb50f77304faa8e86411eae7a11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.168375 1951 tablet_bootstrap.cc:492] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:06.168573 2122 tablet_bootstrap.cc:492] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:06.169282 1951 tablet_bootstrap.cc:654] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.169517 2122 tablet_bootstrap.cc:654] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.170549 2134 tablet_bootstrap.cc:492] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:06.171497 2134 tablet_bootstrap.cc:654] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.171865 1951 tablet_bootstrap.cc:492] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:06.171934 1951 ts_tablet_manager.cc:1367] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:57:06.172387 1951 raft_consensus.cc:348] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.172511 1951 raft_consensus.cc:374] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.172544 1951 raft_consensus.cc:729] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.172648 1951 consensus_queue.cc:260] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.172943 1951 ts_tablet_manager.cc:1398] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:06.173086 2122 tablet_bootstrap.cc:492] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:06.173168 2122 ts_tablet_manager.cc:1367] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:06.173637 2134 tablet_bootstrap.cc:492] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:06.173584 2122 raft_consensus.cc:348] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.173686 2122 raft_consensus.cc:374] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.173696 2134 ts_tablet_manager.cc:1367] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:06.173717 2122 raft_consensus.cc:729] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.173828 2122 consensus_queue.cc:260] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.174090 2122 ts_tablet_manager.cc:1398] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:06.174118 2134 raft_consensus.cc:348] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.174212 2134 raft_consensus.cc:374] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.174242 2134 raft_consensus.cc:729] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.174343 2134 consensus_queue.cc:260] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.174598 2134 ts_tablet_manager.cc:1398] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:06.208957 1760 raft_consensus.cc:482] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:06.209070 1760 raft_consensus.cc:504] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.209444 1760 leader_election.cc:288] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:06.209789 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032eb50f77304faa8e86411eae7a11a0" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:06.209928 24725 raft_consensus.cc:2454] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:06.209980 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032eb50f77304faa8e86411eae7a11a0" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:06.210120 27163 raft_consensus.cc:2454] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:06.210218 24529 leader_election.cc:302] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:06.210361 1760 raft_consensus.cc:2781] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:06.210430 1760 raft_consensus.cc:482] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:06.210464 1760 raft_consensus.cc:3037] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.211515 1760 raft_consensus.cc:504] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.211838 1760 leader_election.cc:288] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:06.212059 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032eb50f77304faa8e86411eae7a11a0" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:06.212150 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032eb50f77304faa8e86411eae7a11a0" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:06.212188 24725 raft_consensus.cc:3037] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.212258 27163 raft_consensus.cc:3037] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.213135 24725 raft_consensus.cc:2454] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:06.213232 27163 raft_consensus.cc:2454] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:06.213447 24529 leader_election.cc:302] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:06.213739 1760 raft_consensus.cc:2781] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:06.213821 1760 raft_consensus.cc:686] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:06.213953 1760 consensus_queue.cc:237] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.216223 24449 catalog_manager.cc:5484] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:06.236510 2196 ts_tablet_manager.cc:902] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:06.237066 2196 tablet_copy_client.cc:308] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:57:06.237692 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet d29130c48f7a435498e854973a090b37 from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:54311)
I20240501 19:57:06.237800 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet d29130c48f7a435498e854973a090b37 from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:54311: session id = 29d1a658e2b946e0b412e5fef82db9d9-d29130c48f7a435498e854973a090b37
I20240501 19:57:06.238809 27052 tablet_copy_source_session.cc:205] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:06.240675 2196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29130c48f7a435498e854973a090b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.244138 2196 tablet_copy_client.cc:783] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:06.244339 2196 tablet_copy_client.cc:647] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:06.246407 2196 tablet_copy_client.cc:523] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:06.248944 2196 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:06.251564 2201 ts_tablet_manager.cc:902] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:06.252204 2201 tablet_copy_client.cc:308] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:57:06.252709 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet f1cd92f84a944b94b5948123e3e1d7a1 from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:38719)
I20240501 19:57:06.252811 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet f1cd92f84a944b94b5948123e3e1d7a1 from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:38719: session id = 53e34bb09d8e41129a5683079ff63610-f1cd92f84a944b94b5948123e3e1d7a1
I20240501 19:57:06.253325 2196 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:06.253520 2196 tablet_bootstrap.cc:492] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:57:06.253628 2196 ts_tablet_manager.cc:1367] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 19:57:06.253830 24746 tablet_copy_source_session.cc:205] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:06.253949 2196 raft_consensus.cc:348] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.254029 2196 raft_consensus.cc:729] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.254130 2196 consensus_queue.cc:260] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:06.257551 2196 ts_tablet_manager.cc:1398] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:57:06.258708 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-d29130c48f7a435498e854973a090b37 received from {username='slave'} at 127.17.65.6:54311
I20240501 19:57:06.258790 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-d29130c48f7a435498e854973a090b37 on tablet d29130c48f7a435498e854973a090b37 with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:06.262495 2201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cd92f84a944b94b5948123e3e1d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.265049 2201 tablet_copy_client.cc:783] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:06.266281 2201 tablet_copy_client.cc:647] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:06.272131 2201 tablet_copy_client.cc:523] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:06.273460 2201 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:06.307142 2201 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:06.307556 27267 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:06.307595 2201 tablet_bootstrap.cc:492] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:57:06.307776 2201 ts_tablet_manager.cc:1367] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.005s
I20240501 19:57:06.308328 2201 raft_consensus.cc:348] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.308578 2201 raft_consensus.cc:729] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.308753 2201 consensus_queue.cc:260] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:06.308948 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:06.310564 2201 ts_tablet_manager.cc:1398] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:06.311338 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-f1cd92f84a944b94b5948123e3e1d7a1 received from {username='slave'} at 127.17.65.7:38719
I20240501 19:57:06.311419 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-f1cd92f84a944b94b5948123e3e1d7a1 on tablet f1cd92f84a944b94b5948123e3e1d7a1 with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:57:06.314460 24836 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:06.323710 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:06.324237 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:06.324280 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:06.324668 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:06.324952 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:06.328871 24725 raft_consensus.cc:1261] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:06.329566 1760 consensus_queue.cc:1035] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.335372 27011 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:06.335940 24449 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 032eb50f77304faa8e86411eae7a11a0 (table rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]). Expected version 1 got 0
I20240501 19:57:06.339215 27163 raft_consensus.cc:1261] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:06.339666 1760 consensus_queue.cc:1035] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.344121 24449 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 032eb50f77304faa8e86411eae7a11a0 (table rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]). Expected version 1 got 0
I20240501 19:57:06.348897 24449 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 032eb50f77304faa8e86411eae7a11a0 (table rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]). Expected version 1 got 0
I20240501 19:57:06.371450 2223 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:06.372059 2223 tablet.cc:1620] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:06.375046 2222 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:06.375443 2222 tablet.cc:1620] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:06.387121 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8] alter complete (version 1)
I20240501 19:57:06.389169 1831 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:06.389468 1831 tablet.cc:1620] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:06.390990 24454 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 032eb50f77304faa8e86411eae7a11a0 (table rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]). Expected version 1 got 0
I20240501 19:57:06.494335 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:06.500375 2222 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:06.500730 2222 tablet.cc:1620] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:06.501894 1831 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:06.502279 1831 tablet.cc:1620] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:06.505475 2223 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:06.505767 2223 tablet.cc:1620] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:06.508484 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8] alter complete (version 2)
I20240501 19:57:06.537672 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:06.541011 24836 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:06.542546 27011 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:06.542897 1859 consensus_queue.cc:1035] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.547950 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:06.551791 1859 consensus_queue.cc:1035] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.556702 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:06.563234 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:06.572403 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:06.587612 27267 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:06.599074 2014 consensus_queue.cc:1035] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.606472 1023 consensus_queue.cc:1035] T e19523b5482d4e5a82252f77779afb6d P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.617650 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:06.623714 2222 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:06.624692 2223 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:06.628726 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8] alter complete (version 3)
I20240501 19:57:06.629669 925 consensus_queue.cc:1035] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.634305 1831 tablet.cc:1722] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:06.647015 925 consensus_queue.cc:1035] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.657377 27162 raft_consensus.cc:1203] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:06.674110 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:06.685283 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 63b6863529174c1ab51bb3cb2827351e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.685578 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b6863529174c1ab51bb3cb2827351e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.685621 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 5faeea757e7a437a9825db4d42bcab4b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.685890 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faeea757e7a437a9825db4d42bcab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.686010 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 63b6863529174c1ab51bb3cb2827351e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.686264 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b6863529174c1ab51bb3cb2827351e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.687697 24705 tablet_service.cc:1467] Processing CreateTablet for tablet b04034e366b74575abac4a266d86db93 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.687980 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04034e366b74575abac4a266d86db93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.688436 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 5faeea757e7a437a9825db4d42bcab4b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.688647 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faeea757e7a437a9825db4d42bcab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.688779 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 63b6863529174c1ab51bb3cb2827351e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.689033 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b6863529174c1ab51bb3cb2827351e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.690526 2288 tablet_bootstrap.cc:492] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:06.691224 2289 tablet_bootstrap.cc:492] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:06.691335 27267 tablet_service.cc:1467] Processing CreateTablet for tablet 5faeea757e7a437a9825db4d42bcab4b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.691488 27405 tablet_service.cc:1467] Processing CreateTablet for tablet b04034e366b74575abac4a266d86db93 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.691581 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faeea757e7a437a9825db4d42bcab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.691759 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04034e366b74575abac4a266d86db93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.693383 2288 tablet_bootstrap.cc:654] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.693759 2291 tablet_bootstrap.cc:492] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:06.694345 27274 tablet_service.cc:1467] Processing CreateTablet for tablet b04034e366b74575abac4a266d86db93 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a999baa61bc54de884283d55cf5b147a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:06.694566 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04034e366b74575abac4a266d86db93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.695730 2290 tablet_bootstrap.cc:492] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:06.695861 2289 tablet_bootstrap.cc:654] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.695941 2291 tablet_bootstrap.cc:654] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.696591 2293 tablet_bootstrap.cc:492] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:06.696815 2290 tablet_bootstrap.cc:654] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.698201 2297 tablet_bootstrap.cc:492] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:06.699519 2297 tablet_bootstrap.cc:654] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.702207 2293 tablet_bootstrap.cc:654] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.702840 2297 tablet_bootstrap.cc:492] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:06.702940 2297 ts_tablet_manager.cc:1367] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:57:06.703526 2297 raft_consensus.cc:348] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.703706 2289 tablet_bootstrap.cc:492] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:06.703720 2297 raft_consensus.cc:374] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.703737 2293 tablet_bootstrap.cc:492] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:06.703766 2297 raft_consensus.cc:729] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.703852 2289 ts_tablet_manager.cc:1367] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:57:06.703943 2293 ts_tablet_manager.cc:1367] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:06.703896 2297 consensus_queue.cc:260] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.704195 2297 ts_tablet_manager.cc:1398] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:06.704355 2289 raft_consensus.cc:348] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.704499 2289 raft_consensus.cc:374] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.704566 2289 raft_consensus.cc:729] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.704753 2289 consensus_queue.cc:260] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.709656 2293 raft_consensus.cc:348] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.709959 2289 ts_tablet_manager.cc:1398] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:06.709998 2293 raft_consensus.cc:374] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.710045 2293 raft_consensus.cc:729] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.710194 2293 consensus_queue.cc:260] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.710768 2289 tablet_bootstrap.cc:492] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:06.710950 2293 ts_tablet_manager.cc:1398] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:06.711834 2288 tablet_bootstrap.cc:492] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:06.711977 2288 ts_tablet_manager.cc:1367] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 19:57:06.712251 2289 tablet_bootstrap.cc:654] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.712677 2288 raft_consensus.cc:348] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.713577 2288 raft_consensus.cc:374] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.713673 2288 raft_consensus.cc:729] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.713913 2288 consensus_queue.cc:260] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.714457 2288 ts_tablet_manager.cc:1398] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:06.714638 2288 tablet_bootstrap.cc:492] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:06.716121 2288 tablet_bootstrap.cc:654] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.716288 2290 tablet_bootstrap.cc:492] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:06.716387 2290 ts_tablet_manager.cc:1367] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 19:57:06.716845 2290 raft_consensus.cc:348] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.717437 2290 raft_consensus.cc:374] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.717455 2291 tablet_bootstrap.cc:492] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:06.717496 2290 raft_consensus.cc:729] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.717569 2291 ts_tablet_manager.cc:1367] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 19:57:06.717633 2290 consensus_queue.cc:260] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.717938 2290 ts_tablet_manager.cc:1398] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:06.718147 2291 raft_consensus.cc:348] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.718267 2291 raft_consensus.cc:374] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.718307 2291 raft_consensus.cc:729] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.718458 2291 consensus_queue.cc:260] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.718827 2291 ts_tablet_manager.cc:1398] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:06.719915 2291 tablet_bootstrap.cc:492] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:06.721177 2291 tablet_bootstrap.cc:654] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.722569 2288 tablet_bootstrap.cc:492] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:06.722654 2288 ts_tablet_manager.cc:1367] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 19:57:06.722810 2291 tablet_bootstrap.cc:492] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:06.722888 2291 ts_tablet_manager.cc:1367] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:06.723248 2288 raft_consensus.cc:348] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.723368 2288 raft_consensus.cc:374] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.723414 2288 raft_consensus.cc:729] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.723443 2291 raft_consensus.cc:348] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.723580 2291 raft_consensus.cc:374] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.723616 2291 raft_consensus.cc:729] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.723554 2288 consensus_queue.cc:260] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.724344 2288 ts_tablet_manager.cc:1398] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:06.723819 2291 consensus_queue.cc:260] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.724956 2291 ts_tablet_manager.cc:1398] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:06.727741 2289 tablet_bootstrap.cc:492] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:06.727843 2289 ts_tablet_manager.cc:1367] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:57:06.728554 2289 raft_consensus.cc:348] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.728724 2289 raft_consensus.cc:374] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.728802 2289 raft_consensus.cc:729] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.729082 2289 consensus_queue.cc:260] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.729452 2289 ts_tablet_manager.cc:1398] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:06.732481 24449 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:06.732602 24449 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:06.735191 24449 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 032eb50f77304faa8e86411eae7a11a0
I20240501 19:57:06.735966 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 032eb50f77304faa8e86411eae7a11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:06 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:06.735963 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 032eb50f77304faa8e86411eae7a11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:06 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:06.735988 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 032eb50f77304faa8e86411eae7a11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:06 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:06.736297 2311 tablet_replica.cc:330] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:06.736327 2310 tablet_replica.cc:330] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:06.736593 2311 raft_consensus.cc:2229] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:06.736855 2311 raft_consensus.cc:2258] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:06.739246 2311 ts_tablet_manager.cc:1875] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:06.739938 2310 raft_consensus.cc:2229] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:06.740116 2310 raft_consensus.cc:2258] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:06.740676 2310 ts_tablet_manager.cc:1875] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:06.743400 2310 ts_tablet_manager.cc:1888] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:06.743513 2310 log.cc:1196] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/032eb50f77304faa8e86411eae7a11a0
I20240501 19:57:06.743849 2310 ts_tablet_manager.cc:1909] T 032eb50f77304faa8e86411eae7a11a0 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:57:06.744589 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 032eb50f77304faa8e86411eae7a11a0 (table rebalancer_dynamic_table_new_name [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]) successfully deleted
I20240501 19:57:06.744889 2309 tablet_replica.cc:330] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:06.744891 2311 ts_tablet_manager.cc:1888] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:06.744966 2311 log.cc:1196] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/032eb50f77304faa8e86411eae7a11a0
I20240501 19:57:06.745077 2309 raft_consensus.cc:2229] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:06.745293 2311 ts_tablet_manager.cc:1909] T 032eb50f77304faa8e86411eae7a11a0 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:57:06.745325 2309 raft_consensus.cc:2258] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:06.745880 2309 ts_tablet_manager.cc:1875] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:06.746027 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 032eb50f77304faa8e86411eae7a11a0 (table rebalancer_dynamic_table_new_name [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]) successfully deleted
I20240501 19:57:06.748155 2309 ts_tablet_manager.cc:1888] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:06.748286 2309 log.cc:1196] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/032eb50f77304faa8e86411eae7a11a0
I20240501 19:57:06.759178 2309 ts_tablet_manager.cc:1909] T 032eb50f77304faa8e86411eae7a11a0 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:57:06.761049 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 032eb50f77304faa8e86411eae7a11a0 (table rebalancer_dynamic_table_new_name [id=68f3ccc43f7d4f818dbfb45c3c0f1ca8]) successfully deleted
I20240501 19:57:06.764844 27425 consensus_queue.cc:237] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:06.767323 24595 raft_consensus.cc:1261] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:06.768262 24987 raft_consensus.cc:1261] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:06.769356 2284 consensus_queue.cc:1035] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:06.770944 2284 consensus_queue.cc:1035] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:06.773046 2284 raft_consensus.cc:2932] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:06.775427 24595 raft_consensus.cc:2932] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:06.775995 24987 raft_consensus.cc:2932] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:06.776582 24449 catalog_manager.cc:5484] T 2fed0e67f44543d09dbeda7100c7e0d6 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:06.779314 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fed0e67f44543d09dbeda7100c7e0d6 with delete_type TABLET_DATA_TOMBSTONED (TS 3b5968cebd0b4294898fdd89b7a5b3a1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:06.791301 2325 tablet_replica.cc:330] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:06.791533 2325 raft_consensus.cc:2229] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:06.791687 2325 raft_consensus.cc:2258] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:06.792070 2325 ts_tablet_manager.cc:1875] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:06.794294 2325 ts_tablet_manager.cc:1888] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:06.794363 2325 log.cc:1196] T 2fed0e67f44543d09dbeda7100c7e0d6 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/2fed0e67f44543d09dbeda7100c7e0d6
I20240501 19:57:06.794879 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 2fed0e67f44543d09dbeda7100c7e0d6 (table rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]) successfully deleted
I20240501 19:57:06.824221 27276 raft_consensus.cc:1203] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:06.836745 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:06.842737 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 4db69f84ffc04ee49181155954b194bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8c41bdd99495420782a3579fe3e9a95e]), partition=
I20240501 19:57:06.842861 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 4db69f84ffc04ee49181155954b194bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8c41bdd99495420782a3579fe3e9a95e]), partition=
I20240501 19:57:06.843015 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db69f84ffc04ee49181155954b194bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.843124 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db69f84ffc04ee49181155954b194bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.843317 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 4db69f84ffc04ee49181155954b194bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8c41bdd99495420782a3579fe3e9a95e]), partition=
I20240501 19:57:06.843551 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db69f84ffc04ee49181155954b194bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:06.845309 2290 tablet_bootstrap.cc:492] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:06.846302 2290 tablet_bootstrap.cc:654] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.847285 2288 tablet_bootstrap.cc:492] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:06.848460 2288 tablet_bootstrap.cc:654] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.848476 2343 tablet_bootstrap.cc:492] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:06.849354 2343 tablet_bootstrap.cc:654] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:06.849723 2288 tablet_bootstrap.cc:492] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:06.849818 2288 ts_tablet_manager.cc:1367] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:06.850756 2288 raft_consensus.cc:348] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:06.850862 2288 raft_consensus.cc:374] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.850889 2288 raft_consensus.cc:729] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.850979 2288 consensus_queue.cc:260] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:06.851342 2290 tablet_bootstrap.cc:492] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:06.851416 2290 ts_tablet_manager.cc:1367] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 19:57:06.851981 2290 raft_consensus.cc:348] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:06.852110 2290 raft_consensus.cc:374] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.852150 2290 raft_consensus.cc:729] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.852278 2290 consensus_queue.cc:260] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:06.852658 2290 ts_tablet_manager.cc:1398] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:06.854749 2288 ts_tablet_manager.cc:1398] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:06.857897 2343 tablet_bootstrap.cc:492] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:06.860791 2343 ts_tablet_manager.cc:1367] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:06.861991 2343 raft_consensus.cc:348] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:06.862179 2343 raft_consensus.cc:374] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:06.862811 2343 raft_consensus.cc:729] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:06.862955 2343 consensus_queue.cc:260] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:06.867954 2343 ts_tablet_manager.cc:1398] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:06.879375 1038 raft_consensus.cc:482] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:06.879494 1038 raft_consensus.cc:504] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.880023 1038 leader_election.cc:288] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:06.880581 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04034e366b74575abac4a266d86db93" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:06.880710 27425 raft_consensus.cc:2454] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:06.880739 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04034e366b74575abac4a266d86db93" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:06.880849 24727 raft_consensus.cc:2454] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:06.881189 27229 leader_election.cc:302] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:06.881433 1038 raft_consensus.cc:2781] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:06.881493 1038 raft_consensus.cc:482] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:06.881531 1038 raft_consensus.cc:3037] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.882716 1038 raft_consensus.cc:504] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.883070 1038 leader_election.cc:288] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:06.883610 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04034e366b74575abac4a266d86db93" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:06.883718 24727 raft_consensus.cc:3037] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.884745 24727 raft_consensus.cc:2454] T b04034e366b74575abac4a266d86db93 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:06.885164 27227 leader_election.cc:302] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:06.885232 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04034e366b74575abac4a266d86db93" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:06.885357 27425 raft_consensus.cc:3037] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.885401 1038 raft_consensus.cc:2781] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:06.885486 1038 raft_consensus.cc:686] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:06.885613 1038 consensus_queue.cc:237] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:06.886546 27425 raft_consensus.cc:2454] T b04034e366b74575abac4a266d86db93 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:06.889072 24452 catalog_manager.cc:5484] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:06.940999 1038 raft_consensus.cc:482] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:06.941128 1038 raft_consensus.cc:504] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.941588 1038 leader_election.cc:288] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:06.942118 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faeea757e7a437a9825db4d42bcab4b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:06.942248 24987 raft_consensus.cc:2454] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:06.942239 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faeea757e7a437a9825db4d42bcab4b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:06.942397 24856 raft_consensus.cc:2454] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:06.942857 27228 leader_election.cc:302] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:06.943289 1038 raft_consensus.cc:2781] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:06.943346 1038 raft_consensus.cc:482] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:06.943382 1038 raft_consensus.cc:3037] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.946593 1038 raft_consensus.cc:504] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.947232 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faeea757e7a437a9825db4d42bcab4b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:06.947341 24987 raft_consensus.cc:3037] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.948381 24987 raft_consensus.cc:2454] T 5faeea757e7a437a9825db4d42bcab4b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:06.948693 27228 leader_election.cc:302] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:06.948843 2149 raft_consensus.cc:2781] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:06.948880 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faeea757e7a437a9825db4d42bcab4b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:06.948990 2149 raft_consensus.cc:686] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:06.949004 24856 raft_consensus.cc:3037] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.949142 2149 consensus_queue.cc:237] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.950217 24856 raft_consensus.cc:2454] T 5faeea757e7a437a9825db4d42bcab4b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:06.947106 1038 leader_election.cc:288] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:06.952307 24450 catalog_manager.cc:5484] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:06.989377 1667 raft_consensus.cc:482] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:06.989504 1667 raft_consensus.cc:504] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.990036 1667 leader_election.cc:288] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:06.990541 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b6863529174c1ab51bb3cb2827351e" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:06.990689 24595 raft_consensus.cc:2454] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:06.990767 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b6863529174c1ab51bb3cb2827351e" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:06.990881 24987 raft_consensus.cc:2454] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:06.991147 24792 leader_election.cc:302] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:06.991297 1667 raft_consensus.cc:2781] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:06.991355 1667 raft_consensus.cc:482] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:06.991395 1667 raft_consensus.cc:3037] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.992350 1667 raft_consensus.cc:504] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.992620 1667 leader_election.cc:288] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:06.993199 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b6863529174c1ab51bb3cb2827351e" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:06.993316 24595 raft_consensus.cc:3037] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.993575 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b6863529174c1ab51bb3cb2827351e" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:06.993675 24987 raft_consensus.cc:3037] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:06.994318 24595 raft_consensus.cc:2454] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:06.994671 24987 raft_consensus.cc:2454] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:06.994709 24791 leader_election.cc:302] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:06.994948 1667 raft_consensus.cc:2781] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:06.995067 1667 raft_consensus.cc:686] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:06.995214 1667 consensus_queue.cc:237] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:06.997541 24450 catalog_manager.cc:5484] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:07.270035 1845 raft_consensus.cc:482] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:07.270156 1845 raft_consensus.cc:504] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:07.270490 1845 leader_election.cc:288] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:07.270942 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db69f84ffc04ee49181155954b194bc" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:07.271117 27032 raft_consensus.cc:2454] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:07.271412 24922 leader_election.cc:302] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:07.271584 1845 raft_consensus.cc:2781] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:07.271642 1845 raft_consensus.cc:482] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:07.271697 1845 raft_consensus.cc:3037] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.271952 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db69f84ffc04ee49181155954b194bc" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:07.272068 24727 raft_consensus.cc:2454] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:07.272884 1845 raft_consensus.cc:504] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:07.273229 1845 leader_election.cc:288] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:07.273548 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db69f84ffc04ee49181155954b194bc" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:07.273573 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db69f84ffc04ee49181155954b194bc" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:07.273685 27032 raft_consensus.cc:3037] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.273696 24727 raft_consensus.cc:3037] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.274658 27032 raft_consensus.cc:2454] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:07.274658 24727 raft_consensus.cc:2454] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:07.274902 24922 leader_election.cc:302] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:07.275079 1845 raft_consensus.cc:2781] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:07.275177 1845 raft_consensus.cc:686] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:07.275303 1845 consensus_queue.cc:237] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:07.277227 24449 catalog_manager.cc:5484] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:07.309058 27143 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:07.311436 27011 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:07.317831 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:07.321717 27274 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:07.326268 24837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:07.332449 24966 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:07.345526 2374 consensus_queue.cc:1035] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:07.357885 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:07.363101 24574 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:07.376423 2149 consensus_queue.cc:1035] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:07.388777 2149 consensus_queue.cc:1035] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:07.391463 1038 consensus_queue.cc:1035] T 5faeea757e7a437a9825db4d42bcab4b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:07.395613 1667 consensus_queue.cc:1035] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:07.400194 925 consensus_queue.cc:1035] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:07.436574 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:07.440943 27032 raft_consensus.cc:1261] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:07.441298 24727 raft_consensus.cc:1261] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:07.441982 731 consensus_queue.cc:1035] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:07.442107 1845 consensus_queue.cc:1035] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:07.447077 24449 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 4db69f84ffc04ee49181155954b194bc (table rebalancer_dynamic_table [id=8c41bdd99495420782a3579fe3e9a95e]). Expected version 1 got 0
I20240501 19:57:07.448242 24449 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 4db69f84ffc04ee49181155954b194bc (table rebalancer_dynamic_table [id=8c41bdd99495420782a3579fe3e9a95e]). Expected version 1 got 0
I20240501 19:57:07.449129 2428 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:07.449353 2425 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:07.449535 2428 tablet.cc:1620] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:07.450336 24449 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 4db69f84ffc04ee49181155954b194bc (table rebalancer_dynamic_table [id=8c41bdd99495420782a3579fe3e9a95e]). Expected version 1 got 0
I20240501 19:57:07.451093 2425 tablet.cc:1620] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:57:07.455847 27163 consensus_queue.cc:237] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:07.456128 2432 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:07.456507 2432 tablet.cc:1620] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:07.457408 27424 raft_consensus.cc:1261] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:07.458021 24856 raft_consensus.cc:1261] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:07.458418 1200 consensus_queue.cc:1035] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:07.458753 1200 consensus_queue.cc:1035] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:07.461058 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=8c41bdd99495420782a3579fe3e9a95e] alter complete (version 1)
I20240501 19:57:07.461567 1200 raft_consensus.cc:2932] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:07.463408 24856 raft_consensus.cc:2932] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:07.463398 27424 raft_consensus.cc:2932] T 92906e09f3474d7987082e0d33304886 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:07.466034 24452 catalog_manager.cc:5484] T 92906e09f3474d7987082e0d33304886 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:07.473055 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet 92906e09f3474d7987082e0d33304886 with delete_type TABLET_DATA_TOMBSTONED (TS 53e34bb09d8e41129a5683079ff63610 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:07.475368 2443 tablet_replica.cc:330] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:07.475536 2443 raft_consensus.cc:2229] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:07.475710 2443 raft_consensus.cc:2258] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:07.476200 2443 ts_tablet_manager.cc:1875] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:07.478369 2443 ts_tablet_manager.cc:1888] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:07.478451 2443 log.cc:1196] T 92906e09f3474d7987082e0d33304886 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/92906e09f3474d7987082e0d33304886
I20240501 19:57:07.479140 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 92906e09f3474d7987082e0d33304886 (table rebalancer_extra_table_4 [id=d0ea2fc512b5434eaba6ba41e6ba01b4]) successfully deleted
I20240501 19:57:07.508287 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:07.515267 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 104d22832ca64b91bf802662ba4ec437 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.515261 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 104d22832ca64b91bf802662ba4ec437 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.515563 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104d22832ca64b91bf802662ba4ec437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.515563 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104d22832ca64b91bf802662ba4ec437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.516495 27404 tablet_service.cc:1467] Processing CreateTablet for tablet b4dabe21c38a4e3fa3c044be1c5ba276 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.516749 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4dabe21c38a4e3fa3c044be1c5ba276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.517969 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 104d22832ca64b91bf802662ba4ec437 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.518180 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104d22832ca64b91bf802662ba4ec437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.518188 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 49bc415748c44bc6a765ccf11ddd7fea (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.518253 2462 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:07.518455 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bc415748c44bc6a765ccf11ddd7fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.519245 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 49bc415748c44bc6a765ccf11ddd7fea (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.519510 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bc415748c44bc6a765ccf11ddd7fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.518188 27142 tablet_service.cc:1467] Processing CreateTablet for tablet b4dabe21c38a4e3fa3c044be1c5ba276 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.520821 24705 tablet_service.cc:1467] Processing CreateTablet for tablet b4dabe21c38a4e3fa3c044be1c5ba276 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.520987 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4dabe21c38a4e3fa3c044be1c5ba276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.521078 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4dabe21c38a4e3fa3c044be1c5ba276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.523094 2463 tablet_bootstrap.cc:492] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:07.523841 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 49bc415748c44bc6a765ccf11ddd7fea (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:07.523944 2464 tablet_bootstrap.cc:492] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:07.524084 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bc415748c44bc6a765ccf11ddd7fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.524261 2463 tablet_bootstrap.cc:654] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.524796 2464 tablet_bootstrap.cc:654] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.525467 2485 tablet_bootstrap.cc:492] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:07.525709 2463 tablet_bootstrap.cc:492] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:07.525895 2463 ts_tablet_manager.cc:1367] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:07.526275 2462 tablet_bootstrap.cc:654] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.526402 2485 tablet_bootstrap.cc:654] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.526436 2463 raft_consensus.cc:348] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.526665 2463 raft_consensus.cc:374] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.526749 2463 raft_consensus.cc:729] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.526973 2463 consensus_queue.cc:260] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.528231 2464 tablet_bootstrap.cc:492] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:07.528370 2464 ts_tablet_manager.cc:1367] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:07.527356 2463 ts_tablet_manager.cc:1398] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:07.529348 2464 raft_consensus.cc:348] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.529518 2464 raft_consensus.cc:374] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.529549 2471 tablet_bootstrap.cc:492] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:07.529594 2464 raft_consensus.cc:729] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.529805 2464 consensus_queue.cc:260] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.530980 2471 tablet_bootstrap.cc:654] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.531925 2464 ts_tablet_manager.cc:1398] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:07.532136 2464 tablet_bootstrap.cc:492] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:07.533550 2464 tablet_bootstrap.cc:654] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.534960 2485 tablet_bootstrap.cc:492] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:07.535125 2485 ts_tablet_manager.cc:1367] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 19:57:07.536459 2485 raft_consensus.cc:348] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.536553 2463 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:07.536649 2485 raft_consensus.cc:374] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.536741 2485 raft_consensus.cc:729] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.536928 2485 consensus_queue.cc:260] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.537357 2485 ts_tablet_manager.cc:1398] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:07.537724 2463 tablet_bootstrap.cc:654] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.539888 2462 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:07.540035 2462 ts_tablet_manager.cc:1367] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 19:57:07.540588 2462 raft_consensus.cc:348] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.540758 2462 raft_consensus.cc:374] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.540884 2462 raft_consensus.cc:729] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.541524 2461 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:07.542587 2462 consensus_queue.cc:260] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.544332 2461 tablet_bootstrap.cc:654] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.544795 2462 ts_tablet_manager.cc:1398] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:07.545369 2464 tablet_bootstrap.cc:492] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:07.545454 2464 ts_tablet_manager.cc:1367] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:57:07.545550 2463 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:07.545634 2463 ts_tablet_manager.cc:1367] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:07.545769 2471 tablet_bootstrap.cc:492] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:07.545843 2471 ts_tablet_manager.cc:1367] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:57:07.545912 2464 raft_consensus.cc:348] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.546015 2464 raft_consensus.cc:374] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.546058 2464 raft_consensus.cc:729] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.546066 2463 raft_consensus.cc:348] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.546185 2464 consensus_queue.cc:260] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.546325 2471 raft_consensus.cc:348] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.546428 2471 raft_consensus.cc:374] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.546463 2471 raft_consensus.cc:729] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.546510 2464 ts_tablet_manager.cc:1398] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:07.547870 2471 consensus_queue.cc:260] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.548758 2471 ts_tablet_manager.cc:1398] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:07.546159 2463 raft_consensus.cc:374] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.549271 2463 raft_consensus.cc:729] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.549487 2463 consensus_queue.cc:260] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.549862 2461 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:07.549957 2461 ts_tablet_manager.cc:1367] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:57:07.550421 2461 raft_consensus.cc:348] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.550522 2461 raft_consensus.cc:374] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.550554 2461 raft_consensus.cc:729] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.550662 2463 ts_tablet_manager.cc:1398] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:07.550662 2461 consensus_queue.cc:260] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.550944 2461 ts_tablet_manager.cc:1398] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:07.551148 2461 tablet_bootstrap.cc:492] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:07.552063 2461 tablet_bootstrap.cc:654] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.555295 2461 tablet_bootstrap.cc:492] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:07.555655 2461 ts_tablet_manager.cc:1367] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:07.556155 2461 raft_consensus.cc:348] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.556401 2461 raft_consensus.cc:374] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.556484 2461 raft_consensus.cc:729] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.556632 2461 consensus_queue.cc:260] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.556939 2461 ts_tablet_manager.cc:1398] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:07.572618 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:07.578292 2425 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:07.578613 2425 tablet.cc:1620] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:57:07.579672 2428 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:07.579965 2428 tablet.cc:1620] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:07.581365 2432 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:07.581635 2432 tablet.cc:1620] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:07.589396 24445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=8c41bdd99495420782a3579fe3e9a95e] alter complete (version 2)
I20240501 19:57:07.617048 2437 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:07.617159 2437 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.617564 2437 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:07.618098 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dabe21c38a4e3fa3c044be1c5ba276" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:07.618254 27424 raft_consensus.cc:2454] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:07.618542 27098 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:07.618561 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dabe21c38a4e3fa3c044be1c5ba276" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:07.618667 24727 raft_consensus.cc:2454] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:07.618700 2437 raft_consensus.cc:2781] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:07.618763 2437 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:07.618803 2437 raft_consensus.cc:3037] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.618867 1038 raft_consensus.cc:482] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:07.618968 1038 raft_consensus.cc:504] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.619385 1038 leader_election.cc:288] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:07.619865 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104d22832ca64b91bf802662ba4ec437" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:07.619974 27424 raft_consensus.cc:2454] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:07.620012 2437 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.620263 27229 leader_election.cc:302] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:07.620332 2437 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:07.620594 2374 raft_consensus.cc:2781] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:07.620646 2374 raft_consensus.cc:482] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:07.620682 2374 raft_consensus.cc:3037] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.620975 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104d22832ca64b91bf802662ba4ec437" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:07.621083 27032 raft_consensus.cc:2454] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:07.621424 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dabe21c38a4e3fa3c044be1c5ba276" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:07.621524 24727 raft_consensus.cc:3037] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.621732 2374 raft_consensus.cc:504] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.621994 2374 leader_election.cc:288] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:07.622524 24727 raft_consensus.cc:2454] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:07.622810 27096 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:07.622920 27425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104d22832ca64b91bf802662ba4ec437" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:07.622987 2437 raft_consensus.cc:2781] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:07.623028 27425 raft_consensus.cc:3037] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.623121 2437 raft_consensus.cc:686] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:07.623267 2437 consensus_queue.cc:237] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.623634 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104d22832ca64b91bf802662ba4ec437" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:07.623739 27032 raft_consensus.cc:3037] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.624090 27425 raft_consensus.cc:2454] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:07.624295 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dabe21c38a4e3fa3c044be1c5ba276" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:07.624382 27423 raft_consensus.cc:3037] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.624797 27032 raft_consensus.cc:2454] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:07.625402 27423 raft_consensus.cc:2454] T b4dabe21c38a4e3fa3c044be1c5ba276 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:07.625679 27227 leader_election.cc:302] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:07.625967 24450 catalog_manager.cc:5484] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:07.626807 2149 raft_consensus.cc:2781] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:07.626883 2149 raft_consensus.cc:686] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:07.626996 2149 consensus_queue.cc:237] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:07.630543 24451 catalog_manager.cc:5484] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:07.691476 24451 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:07.697682 2425 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:07.698154 2428 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:07.698904 2432 tablet.cc:1722] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:07.703474 24451 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=8c41bdd99495420782a3579fe3e9a95e] alter complete (version 3)
I20240501 19:57:07.722318 2018 raft_consensus.cc:482] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:07.722450 2018 raft_consensus.cc:504] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.722839 2018 leader_election.cc:288] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:07.723134 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bc415748c44bc6a765ccf11ddd7fea" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:07.723155 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bc415748c44bc6a765ccf11ddd7fea" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:07.723270 27162 raft_consensus.cc:2454] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:07.723270 24856 raft_consensus.cc:2454] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:07.723523 26967 leader_election.cc:302] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:57:07.723716 2018 raft_consensus.cc:2781] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:07.723771 2018 raft_consensus.cc:482] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:07.723807 2018 raft_consensus.cc:3037] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.724936 2018 raft_consensus.cc:504] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.725255 2018 leader_election.cc:288] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:07.725659 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bc415748c44bc6a765ccf11ddd7fea" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:07.725706 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bc415748c44bc6a765ccf11ddd7fea" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:07.725776 24856 raft_consensus.cc:3037] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.725791 27162 raft_consensus.cc:3037] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.726763 27162 raft_consensus.cc:2454] T 49bc415748c44bc6a765ccf11ddd7fea P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:07.726763 24856 raft_consensus.cc:2454] T 49bc415748c44bc6a765ccf11ddd7fea P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:07.727129 26966 leader_election.cc:302] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:07.727396 2018 raft_consensus.cc:2781] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:07.728089 2018 raft_consensus.cc:686] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:07.728210 2018 consensus_queue.cc:237] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.730443 24450 catalog_manager.cc:5484] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:07.757351 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:07.764981 27142 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:07.769593 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:07.777743 27274 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:07.778643 24574 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:07.780864 24966 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:07.806867 24450 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:07.807021 24450 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:07.809226 24450 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 4db69f84ffc04ee49181155954b194bc
I20240501 19:57:07.809839 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db69f84ffc04ee49181155954b194bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:07 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:07.810173 2518 tablet_replica.cc:330] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:07.810236 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db69f84ffc04ee49181155954b194bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:07 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:07.810317 2518 raft_consensus.cc:2229] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:07.810444 2518 raft_consensus.cc:2258] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:07.810935 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db69f84ffc04ee49181155954b194bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:07 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:07.814417 24837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:07.820180 2520 tablet_replica.cc:330] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:07.820358 2520 raft_consensus.cc:2229] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:07.820605 2520 raft_consensus.cc:2258] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:07.821138 27011 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:07.821215 2520 ts_tablet_manager.cc:1875] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:07.825421 2520 ts_tablet_manager.cc:1888] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:07.825567 2520 log.cc:1196] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/4db69f84ffc04ee49181155954b194bc
I20240501 19:57:07.825984 2520 ts_tablet_manager.cc:1909] T 4db69f84ffc04ee49181155954b194bc P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:57:07.827136 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 4db69f84ffc04ee49181155954b194bc (table rebalancer_dynamic_table_new_name [id=8c41bdd99495420782a3579fe3e9a95e]) successfully deleted
I20240501 19:57:07.828987 2519 tablet_replica.cc:330] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:07.830086 2519 raft_consensus.cc:2229] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:07.830324 2519 raft_consensus.cc:2258] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:07.831633 2519 ts_tablet_manager.cc:1875] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:07.838205 2519 ts_tablet_manager.cc:1888] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:07.838644 2518 ts_tablet_manager.cc:1875] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:07.840874 2518 ts_tablet_manager.cc:1888] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:07.841224 2518 log.cc:1196] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/4db69f84ffc04ee49181155954b194bc
I20240501 19:57:07.841591 2518 ts_tablet_manager.cc:1909] T 4db69f84ffc04ee49181155954b194bc P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:57:07.842142 2519 log.cc:1196] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/4db69f84ffc04ee49181155954b194bc
I20240501 19:57:07.842329 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 4db69f84ffc04ee49181155954b194bc (table rebalancer_dynamic_table_new_name [id=8c41bdd99495420782a3579fe3e9a95e]) successfully deleted
I20240501 19:57:07.842399 2519 ts_tablet_manager.cc:1909] T 4db69f84ffc04ee49181155954b194bc P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:07.842985 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 4db69f84ffc04ee49181155954b194bc (table rebalancer_dynamic_table_new_name [id=8c41bdd99495420782a3579fe3e9a95e]) successfully deleted
I20240501 19:57:07.873392 24595 consensus_queue.cc:237] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:07.874496 27162 raft_consensus.cc:1261] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:07.874518 24986 raft_consensus.cc:1261] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:07.874879 2304 consensus_queue.cc:1035] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:07.875089 1202 consensus_queue.cc:1035] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:07.877065 2539 raft_consensus.cc:2932] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:07.877434 27162 raft_consensus.cc:2932] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:07.877746 24986 raft_consensus.cc:2932] T 5a37dfc7a4324e2b8c43acb51d07a277 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:07.880630 24450 catalog_manager.cc:5484] T 5a37dfc7a4324e2b8c43acb51d07a277 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:07.884090 24837 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a37dfc7a4324e2b8c43acb51d07a277 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:07.884649 2540 tablet_replica.cc:330] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:07.884874 2540 raft_consensus.cc:2229] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:07.885092 2540 raft_consensus.cc:2258] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:07.885643 2540 ts_tablet_manager.cc:1875] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:07.888033 2540 ts_tablet_manager.cc:1888] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:07.888144 2540 log.cc:1196] T 5a37dfc7a4324e2b8c43acb51d07a277 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/5a37dfc7a4324e2b8c43acb51d07a277
I20240501 19:57:07.888796 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 5a37dfc7a4324e2b8c43acb51d07a277 (table rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]) successfully deleted
I20240501 19:57:07.910815 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:07.916564 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8e4102e03f42d7811a7d86bfb5668d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa8e846ca16949c5946f6b8df218f4b6]), partition=
I20240501 19:57:07.916819 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8e4102e03f42d7811a7d86bfb5668d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.916788 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8e4102e03f42d7811a7d86bfb5668d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa8e846ca16949c5946f6b8df218f4b6]), partition=
I20240501 19:57:07.916860 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8e4102e03f42d7811a7d86bfb5668d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa8e846ca16949c5946f6b8df218f4b6]), partition=
I20240501 19:57:07.917032 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8e4102e03f42d7811a7d86bfb5668d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.917081 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8e4102e03f42d7811a7d86bfb5668d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.919588 2485 tablet_bootstrap.cc:492] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:07.919721 2566 tablet_bootstrap.cc:492] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:07.919806 2567 tablet_bootstrap.cc:492] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:07.920477 2485 tablet_bootstrap.cc:654] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.920718 2567 tablet_bootstrap.cc:654] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.920837 2566 tablet_bootstrap.cc:654] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.921473 2485 tablet_bootstrap.cc:492] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:07.921555 2485 ts_tablet_manager.cc:1367] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:57:07.922020 2485 raft_consensus.cc:348] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:07.922120 2485 raft_consensus.cc:374] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.922178 2485 raft_consensus.cc:729] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.922308 2485 consensus_queue.cc:260] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:07.922396 2567 tablet_bootstrap.cc:492] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:07.922487 2567 ts_tablet_manager.cc:1367] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:07.922610 2485 ts_tablet_manager.cc:1398] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:07.923082 2567 raft_consensus.cc:348] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:07.923179 2567 raft_consensus.cc:374] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.923208 2567 raft_consensus.cc:729] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.923316 2567 consensus_queue.cc:260] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:07.923626 2567 ts_tablet_manager.cc:1398] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:07.924187 2566 tablet_bootstrap.cc:492] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:07.924273 2566 ts_tablet_manager.cc:1367] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:07.924692 2566 raft_consensus.cc:348] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:07.924827 2566 raft_consensus.cc:374] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.924883 2566 raft_consensus.cc:729] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.925057 2566 consensus_queue.cc:260] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:07.925427 2566 ts_tablet_manager.cc:1398] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:08.002100 1200 consensus_queue.cc:1035] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.005681 2437 consensus_queue.cc:1035] T b4dabe21c38a4e3fa3c044be1c5ba276 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.076941 2149 consensus_queue.cc:1035] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.080355 2149 consensus_queue.cc:1035] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.122956 2399 raft_consensus.cc:482] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:08.123211 2399 raft_consensus.cc:504] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:08.123646 2399 leader_election.cc:288] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:08.123989 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8e4102e03f42d7811a7d86bfb5668d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:08.124154 24987 raft_consensus.cc:2454] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:08.124208 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8e4102e03f42d7811a7d86bfb5668d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:08.124336 24595 raft_consensus.cc:2454] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:08.124444 24661 leader_election.cc:302] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:08.124630 2399 raft_consensus.cc:2781] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:08.124701 2399 raft_consensus.cc:482] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:08.124738 2399 raft_consensus.cc:3037] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.125773 2399 raft_consensus.cc:504] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:08.126057 2399 leader_election.cc:288] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:08.126335 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8e4102e03f42d7811a7d86bfb5668d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:08.126335 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8e4102e03f42d7811a7d86bfb5668d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:08.126441 24987 raft_consensus.cc:3037] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.126463 24595 raft_consensus.cc:3037] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.127491 24987 raft_consensus.cc:2454] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:08.127491 24595 raft_consensus.cc:2454] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:08.127775 24661 leader_election.cc:302] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:08.128285 2399 raft_consensus.cc:2781] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:08.128381 2399 raft_consensus.cc:686] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:08.128526 2399 consensus_queue.cc:237] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:08.130431 24445 catalog_manager.cc:5484] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:08.156178 27011 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:08.156872 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:08.158372 27142 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:08.162489 24574 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:08.168020 27274 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:08.168540 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:08.168766 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:08.195636 24837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:08.260982 27424 consensus_queue.cc:237] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:08.262136 24857 raft_consensus.cc:1261] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:08.262389 27278 raft_consensus.cc:1261] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:08.262540 2517 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:08.262820 1023 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:08.264822 2649 raft_consensus.cc:2932] T 4a4c18dfded64901bb4645af3dbe2dbc P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:08.265188 24857 raft_consensus.cc:2932] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:08.265198 27278 raft_consensus.cc:2932] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:08.267928 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:08.268363 24445 catalog_manager.cc:5484] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 2 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New cstate: current_term: 2 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:08.272962 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a4c18dfded64901bb4645af3dbe2dbc with delete_type TABLET_DATA_TOMBSTONED (TS 29d1a658e2b946e0b412e5fef82db9d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:08.273134 24595 raft_consensus.cc:1261] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.273526 24987 raft_consensus.cc:1261] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.273528 2398 consensus_queue.cc:1035] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.273950 2398 consensus_queue.cc:1035] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.273957 2018 consensus_queue.cc:1035] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.277230 2652 tablet_replica.cc:330] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:08.277361 24451 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 6d8e4102e03f42d7811a7d86bfb5668d (table rebalancer_dynamic_table [id=aa8e846ca16949c5946f6b8df218f4b6]). Expected version 1 got 0
I20240501 19:57:08.278550 2652 raft_consensus.cc:2229] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:08.278735 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:08.278851 2652 raft_consensus.cc:2258] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.279321 2652 ts_tablet_manager.cc:1875] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:08.280324 2495 consensus_queue.cc:1035] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.281805 2652 ts_tablet_manager.cc:1888] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 19:57:08.281909 2652 log.cc:1196] T 4a4c18dfded64901bb4645af3dbe2dbc P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/4a4c18dfded64901bb4645af3dbe2dbc
I20240501 19:57:08.282702 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 4a4c18dfded64901bb4645af3dbe2dbc (table rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]) successfully deleted
I20240501 19:57:08.284106 24445 catalog_manager.cc:5576] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261) does not have the latest schema for tablet 6d8e4102e03f42d7811a7d86bfb5668d (table rebalancer_dynamic_table [id=aa8e846ca16949c5946f6b8df218f4b6]). Expected version 1 got 0
I20240501 19:57:08.285075 2666 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:08.285375 2666 tablet.cc:1620] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:08.288663 24837 tablet_service.cc:1467] Processing CreateTablet for tablet a6e4eb0c0ac04367b54e3ad348591f33 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.288928 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e4eb0c0ac04367b54e3ad348591f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.289081 27274 tablet_service.cc:1467] Processing CreateTablet for tablet cdfc4bc6b3b1475e81c2d2b555fcea69 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.289340 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfc4bc6b3b1475e81c2d2b555fcea69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.290977 2671 tablet_bootstrap.cc:492] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:08.291167 27011 tablet_service.cc:1467] Processing CreateTablet for tablet cdfc4bc6b3b1475e81c2d2b555fcea69 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.291376 2672 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:08.291433 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfc4bc6b3b1475e81c2d2b555fcea69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.291798 27142 tablet_service.cc:1467] Processing CreateTablet for tablet cdfc4bc6b3b1475e81c2d2b555fcea69 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.292032 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfc4bc6b3b1475e81c2d2b555fcea69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.292466 2672 tablet_bootstrap.cc:654] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.293484 2672 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:08.293565 2672 ts_tablet_manager.cc:1367] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:08.293875 27010 tablet_service.cc:1467] Processing CreateTablet for tablet 687340001b144f3e93004348ea7f66ab (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.294086 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687340001b144f3e93004348ea7f66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.294018 2672 raft_consensus.cc:348] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.294129 2672 raft_consensus.cc:374] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.294157 2672 raft_consensus.cc:729] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.294271 2676 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:08.294267 2672 consensus_queue.cc:260] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.294340 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 687340001b144f3e93004348ea7f66ab (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.294524 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687340001b144f3e93004348ea7f66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.294566 2672 ts_tablet_manager.cc:1398] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:08.295717 2676 tablet_bootstrap.cc:654] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.296208 24574 tablet_service.cc:1467] Processing CreateTablet for tablet a6e4eb0c0ac04367b54e3ad348591f33 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.296465 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e4eb0c0ac04367b54e3ad348591f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.297724 2671 tablet_bootstrap.cc:654] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.298240 2566 tablet_bootstrap.cc:492] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:08.299178 2676 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:08.299299 2566 tablet_bootstrap.cc:654] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.299894 2676 ts_tablet_manager.cc:1367] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:08.300164 24966 tablet_service.cc:1467] Processing CreateTablet for tablet a6e4eb0c0ac04367b54e3ad348591f33 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.300295 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=aa8e846ca16949c5946f6b8df218f4b6] alter complete (version 1)
I20240501 19:57:08.300437 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e4eb0c0ac04367b54e3ad348591f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.300473 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 687340001b144f3e93004348ea7f66ab (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.300709 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687340001b144f3e93004348ea7f66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.301379 2566 tablet_bootstrap.cc:492] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:08.301457 2566 ts_tablet_manager.cc:1367] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:08.302000 2566 raft_consensus.cc:348] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.302100 2566 raft_consensus.cc:374] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.302137 2566 raft_consensus.cc:729] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.302265 2566 consensus_queue.cc:260] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.302531 2566 ts_tablet_manager.cc:1398] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:08.302959 2567 tablet_bootstrap.cc:492] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:08.303803 2567 tablet_bootstrap.cc:654] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.305644 2676 raft_consensus.cc:348] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.305756 2676 raft_consensus.cc:374] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.305790 2676 raft_consensus.cc:729] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.305920 2676 consensus_queue.cc:260] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.307269 2676 ts_tablet_manager.cc:1398] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:08.307505 2676 tablet_bootstrap.cc:492] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:08.308490 2676 tablet_bootstrap.cc:654] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.308707 2674 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:08.309286 2663 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:08.309738 2671 tablet_bootstrap.cc:492] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:08.309821 2671 ts_tablet_manager.cc:1367] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 19:57:08.310297 2671 raft_consensus.cc:348] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.310439 2671 raft_consensus.cc:374] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.310475 2671 raft_consensus.cc:729] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.310598 2671 consensus_queue.cc:260] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.310868 2671 ts_tablet_manager.cc:1398] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:08.311939 2663 tablet.cc:1620] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:08.313886 2567 tablet_bootstrap.cc:492] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:08.313971 2567 ts_tablet_manager.cc:1367] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:08.314299 2676 tablet_bootstrap.cc:492] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:08.314381 2676 ts_tablet_manager.cc:1367] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:08.314471 2567 raft_consensus.cc:348] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.314579 2567 raft_consensus.cc:374] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.314610 2567 raft_consensus.cc:729] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.314728 2567 consensus_queue.cc:260] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.314873 2676 raft_consensus.cc:348] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.314980 2674 tablet_bootstrap.cc:654] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.314989 2676 raft_consensus.cc:374] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.315026 2567 ts_tablet_manager.cc:1398] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:08.315025 2676 raft_consensus.cc:729] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.315240 2676 consensus_queue.cc:260] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.315505 2676 ts_tablet_manager.cc:1398] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:08.318214 2672 tablet_bootstrap.cc:492] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:08.317745 2674 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:08.318641 2674 ts_tablet_manager.cc:1367] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:57:08.319172 2672 tablet_bootstrap.cc:654] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.319126 2674 raft_consensus.cc:348] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.319227 2674 raft_consensus.cc:374] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.319260 2674 raft_consensus.cc:729] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.320079 2674 consensus_queue.cc:260] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.320360 2674 ts_tablet_manager.cc:1398] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:08.320544 2674 tablet_bootstrap.cc:492] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:08.321449 2437 raft_consensus.cc:482] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:08.321530 2437 raft_consensus.cc:504] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.321910 2437 leader_election.cc:288] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:08.322321 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687340001b144f3e93004348ea7f66ab" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:08.322408 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687340001b144f3e93004348ea7f66ab" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
W20240501 19:57:08.322846 27096 leader_election.cc:341] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:57:08.323083 27098 leader_election.cc:341] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:08.323156 27098 leader_election.cc:302] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610
I20240501 19:57:08.323882 2672 tablet_bootstrap.cc:492] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:08.323954 2672 ts_tablet_manager.cc:1367] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 19:57:08.324236 2437 raft_consensus.cc:2726] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:08.324399 2672 raft_consensus.cc:348] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.324501 2672 raft_consensus.cc:374] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.324537 2672 raft_consensus.cc:729] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.324649 2672 consensus_queue.cc:260] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.324939 2672 ts_tablet_manager.cc:1398] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:08.327349 2674 tablet_bootstrap.cc:654] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.330328 24445 catalog_manager.cc:5576] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261) does not have the latest schema for tablet 6d8e4102e03f42d7811a7d86bfb5668d (table rebalancer_dynamic_table [id=aa8e846ca16949c5946f6b8df218f4b6]). Expected version 1 got 0
I20240501 19:57:08.331286 2674 tablet_bootstrap.cc:492] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:08.331409 2674 ts_tablet_manager.cc:1367] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:08.332072 2674 raft_consensus.cc:348] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.332177 2674 raft_consensus.cc:374] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.332237 2674 raft_consensus.cc:729] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.332379 2674 consensus_queue.cc:260] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.332736 2674 ts_tablet_manager.cc:1398] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:08.338135 2690 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:08.338455 2690 tablet.cc:1620] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:57:08.414577 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:08.423152 2663 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:08.423514 2663 tablet.cc:1620] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:08.425002 2690 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:08.425302 2690 tablet.cc:1620] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:57:08.425292 2666 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:08.425575 2666 tablet.cc:1620] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:08.432382 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=aa8e846ca16949c5946f6b8df218f4b6] alter complete (version 2)
I20240501 19:57:08.456389 1038 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:08.456490 1038 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.456790 1038 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:08.457083 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:08.457135 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:08.457213 27031 raft_consensus.cc:2454] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:08.457266 27162 raft_consensus.cc:2454] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:08.457477 27227 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:08.457682 1038 raft_consensus.cc:2781] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:08.457732 1038 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:08.457775 1038 raft_consensus.cc:3037] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.458559 1038 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.458854 1038 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:08.459136 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:08.459251 27031 raft_consensus.cc:3037] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.459239 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:08.459344 27162 raft_consensus.cc:3037] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.460318 27031 raft_consensus.cc:2454] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:08.460335 27162 raft_consensus.cc:2454] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:08.460590 27229 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:08.460789 1038 raft_consensus.cc:2781] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:08.460868 1038 raft_consensus.cc:686] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:08.460990 1038 consensus_queue.cc:237] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:08.462918 24452 catalog_manager.cc:5484] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:08.534471 24452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:08.540477 2663 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:08.541180 2666 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:08.541204 2690 tablet.cc:1722] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:08.546949 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=aa8e846ca16949c5946f6b8df218f4b6] alter complete (version 3)
I20240501 19:57:08.560752 1038 raft_consensus.cc:482] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:08.560850 1038 raft_consensus.cc:504] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.561175 1038 leader_election.cc:288] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:08.561432 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687340001b144f3e93004348ea7f66ab" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:08.561481 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687340001b144f3e93004348ea7f66ab" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:08.561550 27163 raft_consensus.cc:2454] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:08.561609 27031 raft_consensus.cc:2454] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:08.561842 27229 leader_election.cc:302] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:08.562045 1038 raft_consensus.cc:2781] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:08.562096 1038 raft_consensus.cc:482] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:08.562175 1038 raft_consensus.cc:3037] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.563176 1038 raft_consensus.cc:504] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.563498 1038 leader_election.cc:288] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:08.563781 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687340001b144f3e93004348ea7f66ab" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:08.563794 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687340001b144f3e93004348ea7f66ab" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:08.563892 27163 raft_consensus.cc:3037] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.563961 27031 raft_consensus.cc:3037] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.564872 27031 raft_consensus.cc:2454] T 687340001b144f3e93004348ea7f66ab P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:08.564872 27163 raft_consensus.cc:2454] T 687340001b144f3e93004348ea7f66ab P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:08.565142 27227 leader_election.cc:302] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:08.565377 1038 raft_consensus.cc:2781] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:08.565522 1038 raft_consensus.cc:686] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:08.565644 1038 consensus_queue.cc:237] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.568634 24445 catalog_manager.cc:5484] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:08.593943 2539 raft_consensus.cc:482] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:08.594043 2539 raft_consensus.cc:504] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.594439 2539 leader_election.cc:288] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:08.595085 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4eb0c0ac04367b54e3ad348591f33" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:08.595105 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4eb0c0ac04367b54e3ad348591f33" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:08.595227 24857 raft_consensus.cc:2454] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:08.595237 24986 raft_consensus.cc:2454] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:08.595497 24530 leader_election.cc:302] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:08.595762 2539 raft_consensus.cc:2781] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:08.595811 2539 raft_consensus.cc:482] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:08.595860 2539 raft_consensus.cc:3037] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.596693 2539 raft_consensus.cc:504] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.596982 2539 leader_election.cc:288] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:08.597261 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4eb0c0ac04367b54e3ad348591f33" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:08.597357 24857 raft_consensus.cc:3037] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.597441 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4eb0c0ac04367b54e3ad348591f33" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:08.597561 24986 raft_consensus.cc:3037] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.598290 24857 raft_consensus.cc:2454] T a6e4eb0c0ac04367b54e3ad348591f33 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:08.598379 24986 raft_consensus.cc:2454] T a6e4eb0c0ac04367b54e3ad348591f33 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:08.598627 24530 leader_election.cc:302] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:08.598872 2539 raft_consensus.cc:2781] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:08.598956 2539 raft_consensus.cc:686] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:08.599098 2539 consensus_queue.cc:237] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:08.601435 24445 catalog_manager.cc:5484] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:08.627178 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:08.627395 27010 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:08.627468 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:08.629524 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:08.637977 27274 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:08.638352 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:08.643401 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:08.643716 24837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:08.650632 24445 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:08.650782 24445 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:08.652993 24445 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 6d8e4102e03f42d7811a7d86bfb5668d
I20240501 19:57:08.653719 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8e4102e03f42d7811a7d86bfb5668d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:08 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:08.655197 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8e4102e03f42d7811a7d86bfb5668d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:08 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:08.655514 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8e4102e03f42d7811a7d86bfb5668d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:08 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:08.655586 2740 tablet_replica.cc:330] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:08.655833 2740 raft_consensus.cc:2229] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:08.656287 2740 raft_consensus.cc:2258] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.656997 2740 ts_tablet_manager.cc:1875] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:08.661913 2741 tablet_replica.cc:330] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:08.662115 2741 raft_consensus.cc:2229] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:08.668897 2741 raft_consensus.cc:2258] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.669610 2741 ts_tablet_manager.cc:1875] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:08.671751 2740 ts_tablet_manager.cc:1888] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:08.671885 2740 log.cc:1196] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d8e4102e03f42d7811a7d86bfb5668d
I20240501 19:57:08.671934 2741 ts_tablet_manager.cc:1888] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:08.672097 2741 log.cc:1196] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/6d8e4102e03f42d7811a7d86bfb5668d
I20240501 19:57:08.672461 2741 ts_tablet_manager.cc:1909] T 6d8e4102e03f42d7811a7d86bfb5668d P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:57:08.672519 2740 ts_tablet_manager.cc:1909] T 6d8e4102e03f42d7811a7d86bfb5668d P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:57:08.673159 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 6d8e4102e03f42d7811a7d86bfb5668d (table rebalancer_dynamic_table_new_name [id=aa8e846ca16949c5946f6b8df218f4b6]) successfully deleted
I20240501 19:57:08.673241 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 6d8e4102e03f42d7811a7d86bfb5668d (table rebalancer_dynamic_table_new_name [id=aa8e846ca16949c5946f6b8df218f4b6]) successfully deleted
I20240501 19:57:08.676141 2739 tablet_replica.cc:330] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:08.679397 2739 raft_consensus.cc:2229] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:08.680151 2739 raft_consensus.cc:2258] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.683645 2739 ts_tablet_manager.cc:1875] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:08.685715 2739 ts_tablet_manager.cc:1888] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:08.685779 2739 log.cc:1196] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d8e4102e03f42d7811a7d86bfb5668d
I20240501 19:57:08.687886 2739 ts_tablet_manager.cc:1909] T 6d8e4102e03f42d7811a7d86bfb5668d P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:57:08.688542 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 6d8e4102e03f42d7811a7d86bfb5668d (table rebalancer_dynamic_table_new_name [id=aa8e846ca16949c5946f6b8df218f4b6]) successfully deleted
I20240501 19:57:08.714378 24986 consensus_queue.cc:237] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:08.715736 27031 raft_consensus.cc:1261] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:08.716096 24857 raft_consensus.cc:1261] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:08.716109 2688 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:08.716480 2688 consensus_queue.cc:1035] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:08.718309 2502 raft_consensus.cc:2932] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:08.718961 27031 raft_consensus.cc:2932] T 0f1f7372e5ce42b284cf1179c72f578d P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:08.719393 24857 raft_consensus.cc:2932] T 0f1f7372e5ce42b284cf1179c72f578d P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:08.723387 24445 catalog_manager.cc:5484] T 0f1f7372e5ce42b284cf1179c72f578d P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 4 to 6, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:08.726328 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1f7372e5ce42b284cf1179c72f578d with delete_type TABLET_DATA_TOMBSTONED (TS 9081292597be4a72a42b216fddcd7971 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:08.726636 2739 tablet_replica.cc:330] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:08.726778 2739 raft_consensus.cc:2229] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:08.726883 2739 raft_consensus.cc:2258] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.727509 2739 ts_tablet_manager.cc:1875] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:08.729749 2739 ts_tablet_manager.cc:1888] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:57:08.729815 2739 log.cc:1196] T 0f1f7372e5ce42b284cf1179c72f578d P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/0f1f7372e5ce42b284cf1179c72f578d
I20240501 19:57:08.730291 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 0f1f7372e5ce42b284cf1179c72f578d (table rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]) successfully deleted
I20240501 19:57:08.755354 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:08.760540 27010 tablet_service.cc:1467] Processing CreateTablet for tablet 21595ce218334e0385549d114fa723e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2622826745540539567a3eaad5140eb]), partition=
I20240501 19:57:08.760800 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21595ce218334e0385549d114fa723e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.762892 2674 tablet_bootstrap.cc:492] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:08.763464 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 21595ce218334e0385549d114fa723e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2622826745540539567a3eaad5140eb]), partition=
I20240501 19:57:08.763690 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21595ce218334e0385549d114fa723e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.763765 2674 tablet_bootstrap.cc:654] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.764782 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 21595ce218334e0385549d114fa723e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2622826745540539567a3eaad5140eb]), partition=
I20240501 19:57:08.765022 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21595ce218334e0385549d114fa723e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.765775 2567 tablet_bootstrap.cc:492] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:08.766598 2567 tablet_bootstrap.cc:654] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.767805 2674 tablet_bootstrap.cc:492] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:08.767885 2674 ts_tablet_manager.cc:1367] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:08.768191 27163 consensus_queue.cc:237] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:08.768306 2674 raft_consensus.cc:348] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.768405 2674 raft_consensus.cc:374] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.768436 2674 raft_consensus.cc:729] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.768498 2567 tablet_bootstrap.cc:492] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:08.768533 2674 consensus_queue.cc:260] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.768569 2567 ts_tablet_manager.cc:1367] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:08.768805 2674 ts_tablet_manager.cc:1398] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:08.769007 2567 raft_consensus.cc:348] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.769114 2567 raft_consensus.cc:374] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.769155 2567 raft_consensus.cc:729] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.769285 2567 consensus_queue.cc:260] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.769565 2567 ts_tablet_manager.cc:1398] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:08.770036 2792 tablet_bootstrap.cc:492] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:08.770654 24986 raft_consensus.cc:1261] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.771714 27423 raft_consensus.cc:1261] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.771988 2437 consensus_queue.cc:1035] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:08.772805 2792 tablet_bootstrap.cc:654] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
W20240501 19:57:08.772866 27096 consensus_peers.cc:485] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b747487a083433d91191aa79458f51e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:08.773808 2792 tablet_bootstrap.cc:492] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:08.773885 2792 ts_tablet_manager.cc:1367] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:08.774003 2437 consensus_queue.cc:1035] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:08.774281 2792 raft_consensus.cc:348] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.774364 2792 raft_consensus.cc:374] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.774390 2792 raft_consensus.cc:729] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.774487 2792 consensus_queue.cc:260] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.774755 2792 ts_tablet_manager.cc:1398] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:08.782955 2795 raft_consensus.cc:2932] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:08.783357 24986 raft_consensus.cc:2932] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:08.785962 24449 catalog_manager.cc:5484] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:08.786599 27423 raft_consensus.cc:2932] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:08.814795 24725 consensus_queue.cc:237] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:08.816287 27031 raft_consensus.cc:1261] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.816552 27423 raft_consensus.cc:1261] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.816695 2398 consensus_queue.cc:1035] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:08.816898 24661 consensus_peers.cc:485] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98850f18f10c4a57aaaf7a4bd9dce783. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:08.817384 2398 consensus_queue.cc:1035] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:08.819295 2723 raft_consensus.cc:2932] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:08.821075 27423 raft_consensus.cc:2932] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:08.821998 27031 raft_consensus.cc:2932] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:08.822350 24453 catalog_manager.cc:5484] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:08.847986 2765 raft_consensus.cc:482] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:08.848098 2765 raft_consensus.cc:504] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.848580 2765 leader_election.cc:288] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:08.848866 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21595ce218334e0385549d114fa723e8" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:08.848991 27423 raft_consensus.cc:2454] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:08.848986 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21595ce218334e0385549d114fa723e8" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:08.849112 24987 raft_consensus.cc:2454] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:08.849279 26967 leader_election.cc:302] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:08.849428 2765 raft_consensus.cc:2781] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:08.849479 2765 raft_consensus.cc:482] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:08.849519 2765 raft_consensus.cc:3037] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.850373 2765 raft_consensus.cc:504] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.850668 2765 leader_election.cc:288] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:08.850924 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21595ce218334e0385549d114fa723e8" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:08.851056 24987 raft_consensus.cc:3037] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.851022 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21595ce218334e0385549d114fa723e8" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:08.851193 27423 raft_consensus.cc:3037] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.851923 27423 raft_consensus.cc:2454] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:08.852038 24987 raft_consensus.cc:2454] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:08.852177 26967 leader_election.cc:302] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:08.852324 2765 raft_consensus.cc:2781] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:08.852396 2765 raft_consensus.cc:686] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:08.852550 2765 consensus_queue.cc:237] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:08.854867 24445 catalog_manager.cc:5484] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:08.939635 24837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:08.940058 2149 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.942750 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:08.945593 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:08.947847 2149 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.949769 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:08.950969 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:08.955204 27274 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:08.957756 27010 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:08.963728 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:08.976718 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:08.989421 27423 raft_consensus.cc:1261] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.989895 2018 consensus_queue.cc:1035] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.992169 24987 raft_consensus.cc:1261] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.992519 2018 consensus_queue.cc:1035] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.995268 24454 catalog_manager.cc:5576] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261) does not have the latest schema for tablet 21595ce218334e0385549d114fa723e8 (table rebalancer_dynamic_table [id=b2622826745540539567a3eaad5140eb]). Expected version 1 got 0
I20240501 19:57:08.995965 24453 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 21595ce218334e0385549d114fa723e8 (table rebalancer_dynamic_table [id=b2622826745540539567a3eaad5140eb]). Expected version 1 got 0
I20240501 19:57:09.001982 24453 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 21595ce218334e0385549d114fa723e8 (table rebalancer_dynamic_table [id=b2622826745540539567a3eaad5140eb]). Expected version 1 got 0
I20240501 19:57:09.004932 2880 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:09.005349 2880 tablet.cc:1620] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:09.011073 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=b2622826745540539567a3eaad5140eb] alter complete (version 1)
I20240501 19:57:09.017144 2881 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:09.018308 2690 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:09.018579 2690 tablet.cc:1620] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:57:09.021394 2881 tablet.cc:1620] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:09.028637 1202 consensus_queue.cc:1035] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.038038 2539 consensus_queue.cc:1035] T a6e4eb0c0ac04367b54e3ad348591f33 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.114245 1038 consensus_queue.cc:1035] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.114346 24446 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:09.122552 1038 consensus_queue.cc:1035] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.123399 2880 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:09.123705 2880 tablet.cc:1620] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:09.128165 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:09.128404 2690 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:09.128712 2690 tablet.cc:1620] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:57:09.136775 24705 tablet_service.cc:1467] Processing CreateTablet for tablet fb536b9da41e453a9e366a3e20f7ecc2 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.136824 24966 tablet_service.cc:1467] Processing CreateTablet for tablet fb536b9da41e453a9e366a3e20f7ecc2 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.137027 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb536b9da41e453a9e366a3e20f7ecc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.137027 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb536b9da41e453a9e366a3e20f7ecc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.137355 24837 tablet_service.cc:1467] Processing CreateTablet for tablet fb536b9da41e453a9e366a3e20f7ecc2 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.137630 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb536b9da41e453a9e366a3e20f7ecc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.138936 2567 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:09.139144 24838 tablet_service.cc:1467] Processing CreateTablet for tablet ee6a321688af41aa8cd97887b60d4f3a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.139192 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 3a24f60b3dc44f28a49157e335acb230 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.139393 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6a321688af41aa8cd97887b60d4f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.140125 2567 tablet_bootstrap.cc:654] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.141075 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=b2622826745540539567a3eaad5140eb] alter complete (version 2)
I20240501 19:57:09.142076 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 3a24f60b3dc44f28a49157e335acb230 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.142338 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a24f60b3dc44f28a49157e335acb230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.143447 2881 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:09.143522 24575 tablet_service.cc:1467] Processing CreateTablet for tablet ee6a321688af41aa8cd97887b60d4f3a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.143769 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6a321688af41aa8cd97887b60d4f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.143765 2881 tablet.cc:1620] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:09.144723 2931 tablet_bootstrap.cc:492] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:09.145771 2931 tablet_bootstrap.cc:654] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.148536 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 3a24f60b3dc44f28a49157e335acb230 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.148536 27143 tablet_service.cc:1467] Processing CreateTablet for tablet ee6a321688af41aa8cd97887b60d4f3a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.148788 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6a321688af41aa8cd97887b60d4f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.148788 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a24f60b3dc44f28a49157e335acb230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.151196 2934 tablet_bootstrap.cc:492] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:09.151279 2933 tablet_bootstrap.cc:492] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:09.152542 2933 tablet_bootstrap.cc:654] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.153347 2934 tablet_bootstrap.cc:654] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.153402 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a24f60b3dc44f28a49157e335acb230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.155736 2567 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:09.155848 2567 ts_tablet_manager.cc:1367] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:57:09.156343 2567 raft_consensus.cc:348] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.156446 2567 raft_consensus.cc:374] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.156453 2933 tablet_bootstrap.cc:492] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:09.156483 2567 raft_consensus.cc:729] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.156479 2934 tablet_bootstrap.cc:492] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:09.156530 2933 ts_tablet_manager.cc:1367] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:09.156562 2934 ts_tablet_manager.cc:1367] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:09.156615 2567 consensus_queue.cc:260] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.156985 2933 raft_consensus.cc:348] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.157094 2933 raft_consensus.cc:374] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.157043 2934 raft_consensus.cc:348] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.157126 2933 raft_consensus.cc:729] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.157145 2934 raft_consensus.cc:374] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.157181 2934 raft_consensus.cc:729] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.157251 2933 consensus_queue.cc:260] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.157286 2934 consensus_queue.cc:260] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.157516 2933 ts_tablet_manager.cc:1398] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:09.157617 2934 ts_tablet_manager.cc:1398] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:09.159193 2567 ts_tablet_manager.cc:1398] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:09.161527 2928 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:09.163723 2928 tablet_bootstrap.cc:654] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.166630 2928 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:09.166724 2928 ts_tablet_manager.cc:1367] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:09.167213 2928 raft_consensus.cc:348] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.167325 2928 raft_consensus.cc:374] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.167362 2928 raft_consensus.cc:729] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.167487 2928 consensus_queue.cc:260] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.167821 2928 ts_tablet_manager.cc:1398] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:09.170548 2930 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:09.172046 2930 tablet_bootstrap.cc:654] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.172153 2931 tablet_bootstrap.cc:492] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:09.172253 2931 ts_tablet_manager.cc:1367] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240501 19:57:09.172778 2931 raft_consensus.cc:348] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.172881 2931 raft_consensus.cc:374] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.172910 2931 raft_consensus.cc:729] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.173019 2931 consensus_queue.cc:260] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.173296 2931 ts_tablet_manager.cc:1398] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:09.173487 2931 tablet_bootstrap.cc:492] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:09.174057 2930 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:09.174140 2930 ts_tablet_manager.cc:1367] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:09.174654 2930 raft_consensus.cc:348] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.174800 2930 raft_consensus.cc:374] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.174834 2930 raft_consensus.cc:729] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.174950 2930 consensus_queue.cc:260] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.176035 2931 tablet_bootstrap.cc:654] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.179443 2930 ts_tablet_manager.cc:1398] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:57:09.179630 2930 tablet_bootstrap.cc:492] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:09.180573 2930 tablet_bootstrap.cc:654] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.192732 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:09.193123 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:09.198128 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:09.198621 24966 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:09.198643 2688 raft_consensus.cc:482] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:09.198839 2688 raft_consensus.cc:504] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.199213 2688 leader_election.cc:288] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:09.199652 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb536b9da41e453a9e366a3e20f7ecc2" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:09.199774 24856 raft_consensus.cc:2454] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:09.200019 24923 leader_election.cc:302] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:09.200393 2688 raft_consensus.cc:2781] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:09.200441 2688 raft_consensus.cc:482] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:09.200488 2688 raft_consensus.cc:3037] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.201520 2688 raft_consensus.cc:504] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.201843 2688 leader_election.cc:288] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:09.202430 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb536b9da41e453a9e366a3e20f7ecc2" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:09.202530 24856 raft_consensus.cc:3037] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.203490 24856 raft_consensus.cc:2454] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:09.203790 24923 leader_election.cc:302] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:09.204648 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb536b9da41e453a9e366a3e20f7ecc2" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:09.204761 24725 raft_consensus.cc:3037] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.205693 24725 raft_consensus.cc:2454] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:09.206637 27010 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:09.207617 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb536b9da41e453a9e366a3e20f7ecc2" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:09.207778 24727 raft_consensus.cc:2362] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1. Re-sending same reply.
I20240501 19:57:09.208166 2688 raft_consensus.cc:2781] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:09.208252 2688 raft_consensus.cc:686] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:09.208386 2688 consensus_queue.cc:237] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:09.211681 24575 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:09.225088 27274 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:09.228608 24836 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:09.235978 24450 catalog_manager.cc:5484] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:09.255481 2931 tablet_bootstrap.cc:492] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:09.255599 2931 ts_tablet_manager.cc:1367] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.000s
I20240501 19:57:09.256078 2931 raft_consensus.cc:348] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.256193 2931 raft_consensus.cc:374] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.256215 2931 raft_consensus.cc:729] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.256331 2931 consensus_queue.cc:260] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.256634 2931 ts_tablet_manager.cc:1398] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:09.257665 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:09.265931 2930 tablet_bootstrap.cc:492] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:09.266034 2930 ts_tablet_manager.cc:1367] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.086s user 0.001s sys 0.001s
I20240501 19:57:09.266485 2930 raft_consensus.cc:348] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.266618 2930 raft_consensus.cc:374] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.266652 2930 raft_consensus.cc:729] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.266770 2930 consensus_queue.cc:260] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.267112 2930 ts_tablet_manager.cc:1398] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:09.267302 2930 tablet_bootstrap.cc:492] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:09.268209 2930 tablet_bootstrap.cc:654] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.268657 2880 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:09.269882 2690 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:09.273602 2881 tablet.cc:1722] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:09.275887 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=b2622826745540539567a3eaad5140eb] alter complete (version 3)
I20240501 19:57:09.282456 2930 tablet_bootstrap.cc:492] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:09.282562 2930 ts_tablet_manager.cc:1367] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:57:09.283020 2930 raft_consensus.cc:348] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.283160 2930 raft_consensus.cc:374] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.283190 2930 raft_consensus.cc:729] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.283289 2930 consensus_queue.cc:260] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.283521 2930 ts_tablet_manager.cc:1398] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:09.298023 27031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d29130c48f7a435498e854973a090b37"
dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40780
I20240501 19:57:09.298164 27031 raft_consensus.cc:595] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:09.340237 2995 ts_tablet_manager.cc:902] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:09.340826 2995 tablet_copy_client.cc:308] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.6:44579
I20240501 19:57:09.341224 27183 tablet_copy_service.cc:140] P 29d1a658e2b946e0b412e5fef82db9d9: Received BeginTabletCopySession request for tablet 8b747487a083433d91191aa79458f51e from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:37587)
I20240501 19:57:09.341337 27183 tablet_copy_service.cc:161] P 29d1a658e2b946e0b412e5fef82db9d9: Beginning new tablet copy session on tablet 8b747487a083433d91191aa79458f51e from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:37587: session id = bbf7735f82ca4f65a5170ac7909faa47-8b747487a083433d91191aa79458f51e
I20240501 19:57:09.342379 27183 tablet_copy_source_session.cc:205] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:09.344393 2995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b747487a083433d91191aa79458f51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.344485 2765 raft_consensus.cc:979] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: : Instructing follower 9d64cdfaeb2d46b29845160e395fc241 to start an election
I20240501 19:57:09.344594 2765 raft_consensus.cc:1067] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Signalling peer 9d64cdfaeb2d46b29845160e395fc241 to start an election
I20240501 19:57:09.345098 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d29130c48f7a435498e854973a090b37"
dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
from {username='slave'} at 127.17.65.5:53857
I20240501 19:57:09.345259 24857 raft_consensus.cc:482] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:09.345319 24857 raft_consensus.cc:3037] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:09.346349 24857 raft_consensus.cc:504] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:09.346885 24857 leader_election.cc:288] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:09.347749 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:09.347863 24594 raft_consensus.cc:3037] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:09.348129 2995 tablet_copy_client.cc:783] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:09.348268 2995 tablet_copy_client.cc:647] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:09.348275 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:09.348379 27162 raft_consensus.cc:3037] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:09.348783 24594 raft_consensus.cc:2454] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 2.
I20240501 19:57:09.349020 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29130c48f7a435498e854973a090b37" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:09.349128 27031 raft_consensus.cc:3032] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:09.349169 27031 raft_consensus.cc:729] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:09.349342 27162 raft_consensus.cc:2454] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 2.
I20240501 19:57:09.349341 27031 consensus_queue.cc:260] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:09.349524 27031 raft_consensus.cc:3037] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:09.350464 27031 raft_consensus.cc:2454] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 2.
I20240501 19:57:09.351400 24791 leader_election.cc:302] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:09.351743 2724 raft_consensus.cc:482] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:09.351816 2724 raft_consensus.cc:504] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.352159 2724 leader_election.cc:288] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:09.352237 2724 raft_consensus.cc:2781] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:09.352308 2724 raft_consensus.cc:686] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 2 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:09.352442 2724 consensus_queue.cc:237] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:09.354998 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6a321688af41aa8cd97887b60d4f3a" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:09.355103 2995 tablet_copy_client.cc:523] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:09.355146 27162 raft_consensus.cc:2454] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:09.356323 24793 leader_election.cc:302] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:09.357282 2995 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:09.357228 24447 catalog_manager.cc:5484] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 1 to 2, leader changed from 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 2 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:09.358789 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6a321688af41aa8cd97887b60d4f3a" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:09.358899 24594 raft_consensus.cc:2454] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:09.359335 2724 raft_consensus.cc:2781] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:09.359396 2724 raft_consensus.cc:482] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:09.359441 2724 raft_consensus.cc:3037] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.360421 2724 raft_consensus.cc:504] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.360757 2724 leader_election.cc:288] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:09.361055 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6a321688af41aa8cd97887b60d4f3a" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:09.361163 24595 raft_consensus.cc:3037] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.361250 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6a321688af41aa8cd97887b60d4f3a" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:09.361344 27162 raft_consensus.cc:3037] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.361546 2995 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:09.361790 2995 tablet_bootstrap.cc:492] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:09.361946 2995 ts_tablet_manager.cc:1367] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 19:57:09.362174 24595 raft_consensus.cc:2454] T ee6a321688af41aa8cd97887b60d4f3a P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:09.362296 27162 raft_consensus.cc:2454] T ee6a321688af41aa8cd97887b60d4f3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:09.362591 24793 leader_election.cc:302] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:09.362757 2724 raft_consensus.cc:2781] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:09.362838 2724 raft_consensus.cc:686] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:09.362968 2724 consensus_queue.cc:237] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.362396 2995 raft_consensus.cc:348] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.363852 2995 raft_consensus.cc:729] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.364053 2995 consensus_queue.cc:260] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.364809 2995 ts_tablet_manager.cc:1398] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:09.365293 27183 tablet_copy_service.cc:342] P 29d1a658e2b946e0b412e5fef82db9d9: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-8b747487a083433d91191aa79458f51e received from {username='slave'} at 127.17.65.1:37587
I20240501 19:57:09.365366 27183 tablet_copy_service.cc:433] P 29d1a658e2b946e0b412e5fef82db9d9: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-8b747487a083433d91191aa79458f51e on tablet 8b747487a083433d91191aa79458f51e with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:09.367245 3016 ts_tablet_manager.cc:902] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:09.369010 3016 tablet_copy_client.cc:308] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:57:09.369012 24447 catalog_manager.cc:5484] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:09.369971 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet 98850f18f10c4a57aaaf7a4bd9dce783 from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:46019)
I20240501 19:57:09.370083 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet 98850f18f10c4a57aaaf7a4bd9dce783 from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:46019: session id = 9d64cdfaeb2d46b29845160e395fc241-98850f18f10c4a57aaaf7a4bd9dce783
I20240501 19:57:09.370925 24746 tablet_copy_source_session.cc:205] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:09.371755 3016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98850f18f10c4a57aaaf7a4bd9dce783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.374547 3016 tablet_copy_client.cc:783] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:09.374689 3016 tablet_copy_client.cc:647] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:09.375715 3016 tablet_copy_client.cc:523] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:09.376930 3016 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:09.382413 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:09.382527 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:09.387727 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 21595ce218334e0385549d114fa723e8
I20240501 19:57:09.388278 27010 tablet_service.cc:1514] Processing DeleteTablet for tablet 21595ce218334e0385549d114fa723e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:09 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:09.388775 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 21595ce218334e0385549d114fa723e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:09 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:09.389103 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 21595ce218334e0385549d114fa723e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:09 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:09.393860 3016 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:09.394129 3016 tablet_bootstrap.cc:492] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:57:09.394309 3016 ts_tablet_manager.cc:1367] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 19:57:09.394790 3016 raft_consensus.cc:348] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.394886 3016 raft_consensus.cc:729] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.395005 3016 consensus_queue.cc:260] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.395332 3016 ts_tablet_manager.cc:1398] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:09.397071 3022 tablet_replica.cc:330] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:09.397217 3022 raft_consensus.cc:2229] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:09.397395 3022 raft_consensus.cc:2258] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:09.397631 3023 tablet_replica.cc:330] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:09.397769 3023 raft_consensus.cc:2229] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:09.397924 3023 raft_consensus.cc:2258] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:09.398604 3023 ts_tablet_manager.cc:1875] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:09.399242 3022 ts_tablet_manager.cc:1875] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:09.401387 3022 ts_tablet_manager.cc:1888] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:09.401458 3022 log.cc:1196] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/21595ce218334e0385549d114fa723e8
I20240501 19:57:09.401721 3022 ts_tablet_manager.cc:1909] T 21595ce218334e0385549d114fa723e8 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:09.404515 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-98850f18f10c4a57aaaf7a4bd9dce783 received from {username='slave'} at 127.17.65.3:46019
I20240501 19:57:09.404629 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-98850f18f10c4a57aaaf7a4bd9dce783 on tablet 98850f18f10c4a57aaaf7a4bd9dce783 with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:09.405038 3023 ts_tablet_manager.cc:1888] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:09.405288 3024 tablet_replica.cc:330] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:09.405498 3024 raft_consensus.cc:2229] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:09.405644 3024 raft_consensus.cc:2258] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:09.406376 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 21595ce218334e0385549d114fa723e8 (table rebalancer_dynamic_table_new_name [id=b2622826745540539567a3eaad5140eb]) successfully deleted
I20240501 19:57:09.407076 3024 ts_tablet_manager.cc:1875] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:09.409176 3023 log.cc:1196] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/21595ce218334e0385549d114fa723e8
I20240501 19:57:09.409502 3023 ts_tablet_manager.cc:1909] T 21595ce218334e0385549d114fa723e8 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:57:09.410094 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 21595ce218334e0385549d114fa723e8 (table rebalancer_dynamic_table_new_name [id=b2622826745540539567a3eaad5140eb]) successfully deleted
I20240501 19:57:09.410249 3024 ts_tablet_manager.cc:1888] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:09.410348 3024 log.cc:1196] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/21595ce218334e0385549d114fa723e8
I20240501 19:57:09.410688 3024 ts_tablet_manager.cc:1909] T 21595ce218334e0385549d114fa723e8 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:57:09.411556 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 21595ce218334e0385549d114fa723e8 (table rebalancer_dynamic_table_new_name [id=b2622826745540539567a3eaad5140eb]) successfully deleted
I20240501 19:57:09.431049 2724 raft_consensus.cc:482] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:09.431149 2724 raft_consensus.cc:504] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.431499 2724 leader_election.cc:288] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:09.432503 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a24f60b3dc44f28a49157e335acb230" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:09.432644 27276 raft_consensus.cc:2454] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:09.432704 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a24f60b3dc44f28a49157e335acb230" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:09.432819 24595 raft_consensus.cc:2454] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:09.433097 24791 leader_election.cc:302] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:09.433255 2724 raft_consensus.cc:2781] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:09.433326 2724 raft_consensus.cc:482] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:09.433362 2724 raft_consensus.cc:3037] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.434363 2724 raft_consensus.cc:504] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.434649 2724 leader_election.cc:288] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:09.434984 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a24f60b3dc44f28a49157e335acb230" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:09.435104 24595 raft_consensus.cc:3037] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.435482 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a24f60b3dc44f28a49157e335acb230" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:09.435585 27276 raft_consensus.cc:3037] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.436098 24595 raft_consensus.cc:2454] T 3a24f60b3dc44f28a49157e335acb230 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:09.436612 27276 raft_consensus.cc:2454] T 3a24f60b3dc44f28a49157e335acb230 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:09.436695 24791 leader_election.cc:302] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:09.437275 2724 raft_consensus.cc:2781] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:09.437353 2724 raft_consensus.cc:686] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:09.437465 2724 consensus_queue.cc:237] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:09.440690 24454 catalog_manager.cc:5484] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:09.464572 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:09.466086 27010 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:09.467298 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:09.482810 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:09.486793 24836 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:09.487098 24966 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:09.489648 24449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:09.495141 24575 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:09.499404 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 29b2641b84ab4ebf81badc59f59bbf95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=559dbe7ed639426192710e0ecfb1f490]), partition=
I20240501 19:57:09.499644 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b2641b84ab4ebf81badc59f59bbf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.501849 2928 tablet_bootstrap.cc:492] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:09.502466 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 29b2641b84ab4ebf81badc59f59bbf95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=559dbe7ed639426192710e0ecfb1f490]), partition=
I20240501 19:57:09.502672 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 29b2641b84ab4ebf81badc59f59bbf95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=559dbe7ed639426192710e0ecfb1f490]), partition=
I20240501 19:57:09.502724 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b2641b84ab4ebf81badc59f59bbf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.502729 2928 tablet_bootstrap.cc:654] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.502878 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b2641b84ab4ebf81badc59f59bbf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.504508 2567 tablet_bootstrap.cc:492] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:09.505215 2931 tablet_bootstrap.cc:492] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:09.505391 2567 tablet_bootstrap.cc:654] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.506217 2931 tablet_bootstrap.cc:654] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.506587 2567 tablet_bootstrap.cc:492] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:09.506670 2567 ts_tablet_manager.cc:1367] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:09.507139 2567 raft_consensus.cc:348] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.507249 2567 raft_consensus.cc:374] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.507287 2567 raft_consensus.cc:729] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.507406 2567 consensus_queue.cc:260] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.507719 2567 ts_tablet_manager.cc:1398] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:09.520887 2931 tablet_bootstrap.cc:492] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:09.520985 2931 ts_tablet_manager.cc:1367] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:57:09.521461 2931 raft_consensus.cc:348] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.521564 2931 raft_consensus.cc:374] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.521592 2931 raft_consensus.cc:729] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.521703 2931 consensus_queue.cc:260] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.521884 2928 tablet_bootstrap.cc:492] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:09.521966 2931 ts_tablet_manager.cc:1398] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:09.522001 2928 ts_tablet_manager.cc:1367] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.001s
I20240501 19:57:09.522197 27274 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:09.522462 2928 raft_consensus.cc:348] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.522610 2928 raft_consensus.cc:374] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:09.522663 2928 raft_consensus.cc:729] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:09.522790 2928 consensus_queue.cc:260] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.523100 2928 ts_tablet_manager.cc:1398] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 19:57:09.606736 2723 raft_consensus.cc:482] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:09.606849 2723 raft_consensus.cc:504] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.607271 2723 leader_election.cc:288] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:09.607740 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b2641b84ab4ebf81badc59f59bbf95" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:09.607872 24987 raft_consensus.cc:2454] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:09.607844 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b2641b84ab4ebf81badc59f59bbf95" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:09.607973 24595 raft_consensus.cc:2454] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:09.608158 24661 leader_election.cc:302] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:09.608347 2723 raft_consensus.cc:2781] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:09.608402 2723 raft_consensus.cc:482] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:09.608446 2723 raft_consensus.cc:3037] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.609463 2723 raft_consensus.cc:504] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.609817 2723 leader_election.cc:288] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:09.610196 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b2641b84ab4ebf81badc59f59bbf95" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:09.610302 24987 raft_consensus.cc:3037] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.611260 24987 raft_consensus.cc:2454] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:09.611521 24661 leader_election.cc:302] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:09.611860 2723 raft_consensus.cc:2781] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:09.611902 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b2641b84ab4ebf81badc59f59bbf95" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:09.611953 2723 raft_consensus.cc:686] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:09.612013 24595 raft_consensus.cc:3037] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:09.612066 2723 consensus_queue.cc:237] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:09.612993 24595 raft_consensus.cc:2454] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:09.615890 24454 catalog_manager.cc:5484] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:09.644186 2502 consensus_queue.cc:1035] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.653823 2688 consensus_queue.cc:1035] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.696259 24836 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:09.699501 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:09.701957 27010 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:09.707376 27274 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:09.714517 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:09.716192 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:09.717685 24575 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:09.723430 24966 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:09.727324 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:09.739473 24986 raft_consensus.cc:1261] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:09.740046 2723 consensus_queue.cc:1035] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.740898 24595 raft_consensus.cc:1261] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:09.744148 24454 catalog_manager.cc:5576] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261) does not have the latest schema for tablet 29b2641b84ab4ebf81badc59f59bbf95 (table rebalancer_dynamic_table [id=559dbe7ed639426192710e0ecfb1f490]). Expected version 1 got 0
I20240501 19:57:09.744813 2398 consensus_queue.cc:1035] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.758939 24454 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 29b2641b84ab4ebf81badc59f59bbf95 (table rebalancer_dynamic_table [id=559dbe7ed639426192710e0ecfb1f490]). Expected version 1 got 0
I20240501 19:57:09.771210 2690 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:09.771534 2690 tablet.cc:1620] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:57:09.773804 24857 raft_consensus.cc:1203] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:09.776511 24446 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 29b2641b84ab4ebf81badc59f59bbf95 (table rebalancer_dynamic_table [id=559dbe7ed639426192710e0ecfb1f490]). Expected version 1 got 0
I20240501 19:57:09.778723 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=559dbe7ed639426192710e0ecfb1f490] alter complete (version 1)
I20240501 19:57:09.780711 3102 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:09.780988 3102 tablet.cc:1620] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:09.781286 3101 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:09.782498 3101 tablet.cc:1620] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:09.810653 24725 consensus_queue.cc:237] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:09.811843 27278 raft_consensus.cc:1261] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:09.811887 24986 raft_consensus.cc:1261] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:09.812402 2398 consensus_queue.cc:1035] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:09.812559 2723 consensus_queue.cc:1035] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:09.814325 2398 raft_consensus.cc:2932] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:09.814637 24986 raft_consensus.cc:2932] T f1cd92f84a944b94b5948123e3e1d7a1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:09.814740 27278 raft_consensus.cc:2932] T f1cd92f84a944b94b5948123e3e1d7a1 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:09.817588 24454 catalog_manager.cc:5484] T f1cd92f84a944b94b5948123e3e1d7a1 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:09.820050 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cd92f84a944b94b5948123e3e1d7a1 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:09.821062 3122 tablet_replica.cc:330] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:09.821328 3122 raft_consensus.cc:2229] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:09.821545 3122 raft_consensus.cc:2258] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:09.822126 3122 ts_tablet_manager.cc:1875] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:09.824024 3122 ts_tablet_manager.cc:1888] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:09.824085 3122 log.cc:1196] T f1cd92f84a944b94b5948123e3e1d7a1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/f1cd92f84a944b94b5948123e3e1d7a1
I20240501 19:57:09.824610 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet f1cd92f84a944b94b5948123e3e1d7a1 (table rebalance_test_table_11 [id=781aeb5e430941fbb20252d74c9d4bc5]) successfully deleted
I20240501 19:57:09.839202 24595 raft_consensus.cc:1203] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:09.873370 27032 raft_consensus.cc:1261] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:09.873807 2724 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:09.883893 24595 raft_consensus.cc:1261] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:09.884286 2724 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:09.887372 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:09.891533 27163 raft_consensus.cc:1261] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:09.893766 3000 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:09.895196 3101 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:09.895517 3101 tablet.cc:1620] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:09.897126 3102 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:09.897411 3102 tablet.cc:1620] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:09.900852 2690 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:09.901161 2690 tablet.cc:1620] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: MemRowSet was empty: no flush needed.
I20240501 19:57:09.910506 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=559dbe7ed639426192710e0ecfb1f490] alter complete (version 2)
I20240501 19:57:09.914710 3176 consensus_queue.cc:1035] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.919818 2724 consensus_queue.cc:1035] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.941864 3176 consensus_queue.cc:1035] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.945073 3000 consensus_queue.cc:1035] T 3a24f60b3dc44f28a49157e335acb230 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:09.968914 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:09.974241 24447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:09.976227 24966 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:09.985548 24575 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b1fb143dc410f8f32b68b369b6147 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.985698 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 0a12749d7e07492c9473fa758c309044 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.985848 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b1fb143dc410f8f32b68b369b6147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.985981 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a12749d7e07492c9473fa758c309044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.986594 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 0a12749d7e07492c9473fa758c309044 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.986680 27267 tablet_service.cc:1467] Processing CreateTablet for tablet c88765fca05c4d31b62db826b51d29c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.986853 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a12749d7e07492c9473fa758c309044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.987108 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88765fca05c4d31b62db826b51d29c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.988312 24968 tablet_service.cc:1467] Processing CreateTablet for tablet c88765fca05c4d31b62db826b51d29c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.988452 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:09.988538 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88765fca05c4d31b62db826b51d29c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.988595 2931 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:09.988858 2567 tablet_bootstrap.cc:492] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:09.989518 2931 tablet_bootstrap.cc:654] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.989876 2567 tablet_bootstrap.cc:654] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:09.990720 27143 tablet_service.cc:1467] Processing CreateTablet for tablet c88765fca05c4d31b62db826b51d29c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.990971 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88765fca05c4d31b62db826b51d29c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.991379 24966 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b1fb143dc410f8f32b68b369b6147 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.992566 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b1fb143dc410f8f32b68b369b6147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.994825 24836 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:09.995985 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:09.997939 27010 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:09.998179 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 0a12749d7e07492c9473fa758c309044 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.998283 27011 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b1fb143dc410f8f32b68b369b6147 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:09.998422 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a12749d7e07492c9473fa758c309044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:09.998546 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b1fb143dc410f8f32b68b369b6147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.003834 3199 tablet_bootstrap.cc:492] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:10.005097 3199 tablet_bootstrap.cc:654] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.008514 3203 tablet_bootstrap.cc:492] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:10.009526 3203 tablet_bootstrap.cc:654] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.010815 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:10.012805 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:10.021806 3101 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:10.022706 3102 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:10.027596 2690 tablet.cc:1722] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:10.030458 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=559dbe7ed639426192710e0ecfb1f490] alter complete (version 3)
I20240501 19:57:10.037873 27274 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:10.069685 3202 tablet_bootstrap.cc:492] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:10.071410 2931 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:10.071529 2931 ts_tablet_manager.cc:1367] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.000s
I20240501 19:57:10.071564 3199 tablet_bootstrap.cc:492] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:10.071564 2567 tablet_bootstrap.cc:492] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:10.071662 3199 ts_tablet_manager.cc:1367] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.000s
I20240501 19:57:10.071662 2567 ts_tablet_manager.cc:1367] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.000s
I20240501 19:57:10.071969 2931 raft_consensus.cc:348] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.072088 2931 raft_consensus.cc:374] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.072125 2931 raft_consensus.cc:729] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.072127 2567 raft_consensus.cc:348] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.072238 2567 raft_consensus.cc:374] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.072269 2567 raft_consensus.cc:729] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.072237 2931 consensus_queue.cc:260] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.072386 2567 consensus_queue.cc:260] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.072504 2931 ts_tablet_manager.cc:1398] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.072726 2567 ts_tablet_manager.cc:1398] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.072909 2567 tablet_bootstrap.cc:492] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:10.073792 3199 raft_consensus.cc:348] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.073928 3199 raft_consensus.cc:374] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.073941 2567 tablet_bootstrap.cc:654] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.073968 3199 raft_consensus.cc:729] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.074131 3199 consensus_queue.cc:260] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.075722 3199 ts_tablet_manager.cc:1398] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:10.075904 3199 tablet_bootstrap.cc:492] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:10.076893 2688 raft_consensus.cc:482] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:10.076972 2688 raft_consensus.cc:504] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.077373 2688 leader_election.cc:288] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:10.078037 2567 tablet_bootstrap.cc:492] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:10.078188 2567 ts_tablet_manager.cc:1367] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:10.078874 3199 tablet_bootstrap.cc:654] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.079646 2567 raft_consensus.cc:348] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.079839 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12749d7e07492c9473fa758c309044" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:10.079874 2567 raft_consensus.cc:374] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.080024 2567 raft_consensus.cc:729] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
W20240501 19:57:10.080191 24922 leader_election.cc:341] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:10.080147 2567 consensus_queue.cc:260] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.080430 3199 tablet_bootstrap.cc:492] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:10.080593 3199 ts_tablet_manager.cc:1367] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:10.080616 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12749d7e07492c9473fa758c309044" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:10.080890 2567 ts_tablet_manager.cc:1398] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:10.081070 2567 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:10.081167 3199 raft_consensus.cc:348] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.081305 3199 raft_consensus.cc:374] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.081355 3199 raft_consensus.cc:729] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.081488 3199 consensus_queue.cc:260] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.081822 3199 ts_tablet_manager.cc:1398] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.081998 2567 tablet_bootstrap.cc:654] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.082108 3202 tablet_bootstrap.cc:654] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
W20240501 19:57:10.081993 24924 leader_election.cc:341] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:10.082801 24924 leader_election.cc:302] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610
I20240501 19:57:10.083112 2688 raft_consensus.cc:2726] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:10.083297 2567 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:10.083360 2567 ts_tablet_manager.cc:1367] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:10.083842 3203 tablet_bootstrap.cc:492] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:10.083799 2567 raft_consensus.cc:348] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.083904 2567 raft_consensus.cc:374] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.083930 3203 ts_tablet_manager.cc:1367] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.000s
I20240501 19:57:10.083940 2567 raft_consensus.cc:729] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.084050 2567 consensus_queue.cc:260] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.084296 2567 ts_tablet_manager.cc:1398] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:10.084400 3203 raft_consensus.cc:348] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.084518 3203 raft_consensus.cc:374] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.084568 3203 raft_consensus.cc:729] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.084688 3203 consensus_queue.cc:260] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.084995 3203 ts_tablet_manager.cc:1398] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.085194 3203 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:10.086046 3202 tablet_bootstrap.cc:492] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:10.086135 3202 ts_tablet_manager.cc:1367] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:57:10.086151 3203 tablet_bootstrap.cc:654] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.086686 3202 raft_consensus.cc:348] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.086791 3202 raft_consensus.cc:374] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.086833 3202 raft_consensus.cc:729] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.087708 3203 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:10.087793 3203 ts_tablet_manager.cc:1367] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:10.088212 3203 raft_consensus.cc:348] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.088482 3203 raft_consensus.cc:374] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.088579 3203 raft_consensus.cc:729] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.088778 3203 consensus_queue.cc:260] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.088856 3202 consensus_queue.cc:260] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.089269 3202 ts_tablet_manager.cc:1398] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:10.089485 3203 ts_tablet_manager.cc:1398] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:10.120200 24725 consensus_queue.cc:237] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:10.121389 27032 raft_consensus.cc:1261] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:10.121389 24856 raft_consensus.cc:1261] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:10.121814 2398 consensus_queue.cc:1035] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:10.122072 3121 consensus_queue.cc:1035] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:10.123481 2398 raft_consensus.cc:2932] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:10.124045 27032 raft_consensus.cc:2932] T 98850f18f10c4a57aaaf7a4bd9dce783 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:10.124240 24856 raft_consensus.cc:2932] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:10.127528 24447 catalog_manager.cc:5484] T 98850f18f10c4a57aaaf7a4bd9dce783 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:10.131546 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 98850f18f10c4a57aaaf7a4bd9dce783 with delete_type TABLET_DATA_TOMBSTONED (TS 4a6e54e86a9949e5970b42e8e32e7561 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:10.131901 3234 tablet_replica.cc:330] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:10.132083 3234 raft_consensus.cc:2229] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:10.132318 3234 raft_consensus.cc:2258] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:10.132822 3234 ts_tablet_manager.cc:1875] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:10.135231 3234 ts_tablet_manager.cc:1888] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:10.135345 3234 log.cc:1196] T 98850f18f10c4a57aaaf7a4bd9dce783 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/98850f18f10c4a57aaaf7a4bd9dce783
I20240501 19:57:10.136235 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 98850f18f10c4a57aaaf7a4bd9dce783 (table rebalancer_extra_table_2 [id=3c75b9e628b24bf082d0ab76051e1468]) successfully deleted
I20240501 19:57:10.136394 24453 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:10.136533 24453 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:10.138492 24453 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 29b2641b84ab4ebf81badc59f59bbf95
I20240501 19:57:10.139235 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b2641b84ab4ebf81badc59f59bbf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:10 UTC) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:10.139513 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b2641b84ab4ebf81badc59f59bbf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:10 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:10.139585 3240 tablet_replica.cc:330] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:10.139588 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b2641b84ab4ebf81badc59f59bbf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:10 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:10.139739 3240 raft_consensus.cc:2229] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:10.139865 3240 raft_consensus.cc:2258] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:10.139914 3242 tablet_replica.cc:330] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:10.140035 3242 raft_consensus.cc:2229] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:10.140245 3242 raft_consensus.cc:2258] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:10.140480 3240 ts_tablet_manager.cc:1875] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:10.141136 3242 ts_tablet_manager.cc:1875] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:10.143586 3240 ts_tablet_manager.cc:1888] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:10.143728 3240 log.cc:1196] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/29b2641b84ab4ebf81badc59f59bbf95
I20240501 19:57:10.143924 3242 ts_tablet_manager.cc:1888] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:10.144006 3242 log.cc:1196] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/29b2641b84ab4ebf81badc59f59bbf95
I20240501 19:57:10.144145 3240 ts_tablet_manager.cc:1909] T 29b2641b84ab4ebf81badc59f59bbf95 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting consensus metadata
I20240501 19:57:10.144354 3242 ts_tablet_manager.cc:1909] T 29b2641b84ab4ebf81badc59f59bbf95 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:57:10.144846 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 29b2641b84ab4ebf81badc59f59bbf95 (table rebalancer_dynamic_table_new_name [id=559dbe7ed639426192710e0ecfb1f490]) successfully deleted
I20240501 19:57:10.144876 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 29b2641b84ab4ebf81badc59f59bbf95 (table rebalancer_dynamic_table_new_name [id=559dbe7ed639426192710e0ecfb1f490]) successfully deleted
I20240501 19:57:10.147944 3241 tablet_replica.cc:330] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:10.148083 3241 raft_consensus.cc:2229] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:10.148195 3241 raft_consensus.cc:2258] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:10.148689 3241 ts_tablet_manager.cc:1875] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:10.150496 3241 ts_tablet_manager.cc:1888] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:10.150573 3241 log.cc:1196] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/29b2641b84ab4ebf81badc59f59bbf95
I20240501 19:57:10.150913 3241 ts_tablet_manager.cc:1909] T 29b2641b84ab4ebf81badc59f59bbf95 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:57:10.151530 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 29b2641b84ab4ebf81badc59f59bbf95 (table rebalancer_dynamic_table_new_name [id=559dbe7ed639426192710e0ecfb1f490]) successfully deleted
I20240501 19:57:10.182597 24594 consensus_queue.cc:237] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:10.183936 27032 raft_consensus.cc:1261] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:10.184051 24986 raft_consensus.cc:1261] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:10.184089 24530 consensus_peers.cc:485] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ca876a9d51e4096955127e7038e2555. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:10.184404 3264 consensus_queue.cc:1035] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:10.185242 2539 consensus_queue.cc:1035] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:10.187103 3264 raft_consensus.cc:2932] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:10.187613 24986 raft_consensus.cc:2932] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:10.191020 24453 catalog_manager.cc:5484] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:10.192678 27032 raft_consensus.cc:2932] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:10.205871 2765 raft_consensus.cc:482] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:10.205981 2765 raft_consensus.cc:504] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.206355 2765 leader_election.cc:288] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:10.206678 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b1fb143dc410f8f32b68b369b6147" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:10.206696 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b1fb143dc410f8f32b68b369b6147" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:10.206791 24986 raft_consensus.cc:2454] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:10.206811 24594 raft_consensus.cc:2454] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:10.207103 26965 leader_election.cc:302] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:10.207307 2765 raft_consensus.cc:2781] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:10.207355 2765 raft_consensus.cc:482] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:10.207396 2765 raft_consensus.cc:3037] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.208212 2765 raft_consensus.cc:504] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.208500 2765 leader_election.cc:288] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:10.208796 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b1fb143dc410f8f32b68b369b6147" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:10.208915 24594 raft_consensus.cc:3037] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.209054 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b1fb143dc410f8f32b68b369b6147" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:10.209148 24986 raft_consensus.cc:3037] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.209789 24594 raft_consensus.cc:2454] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:10.210072 24986 raft_consensus.cc:2454] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:10.210472 26966 leader_election.cc:302] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:10.210615 2765 raft_consensus.cc:2781] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:10.210712 2765 raft_consensus.cc:686] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:10.210825 2765 consensus_queue.cc:237] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:10.213277 24453 catalog_manager.cc:5484] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:10.226162 24986 consensus_queue.cc:237] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
W20240501 19:57:10.227875 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc23e8b5ad8a4911ae766492fb823806. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:10.227856 24856 raft_consensus.cc:1261] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:10.228298 2688 consensus_queue.cc:1035] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:10.228462 24725 raft_consensus.cc:1261] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:10.228804 2688 consensus_queue.cc:1035] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:10.230475 2502 raft_consensus.cc:2932] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:10.230916 24725 raft_consensus.cc:2932] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:10.232916 24856 raft_consensus.cc:2932] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:10.236078 24453 catalog_manager.cc:5484] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:10.240700 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:10.246248 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 9fed5eb90e034c0dbf49880050c0c935 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c4db455d9504897b3b366b7178f9d73]), partition=
I20240501 19:57:10.246399 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 9fed5eb90e034c0dbf49880050c0c935 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c4db455d9504897b3b366b7178f9d73]), partition=
I20240501 19:57:10.246528 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fed5eb90e034c0dbf49880050c0c935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.246632 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fed5eb90e034c0dbf49880050c0c935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.248759 2931 tablet_bootstrap.cc:492] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:10.248998 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 9fed5eb90e034c0dbf49880050c0c935 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c4db455d9504897b3b366b7178f9d73]), partition=
I20240501 19:57:10.249105 3298 tablet_bootstrap.cc:492] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:10.249229 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fed5eb90e034c0dbf49880050c0c935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.249714 2931 tablet_bootstrap.cc:654] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.250249 3298 tablet_bootstrap.cc:654] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.250861 2931 tablet_bootstrap.cc:492] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:10.250942 2931 ts_tablet_manager.cc:1367] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:10.251420 3298 tablet_bootstrap.cc:492] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:10.251420 3202 tablet_bootstrap.cc:492] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:10.251425 2931 raft_consensus.cc:348] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.251499 3298 ts_tablet_manager.cc:1367] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:10.251534 2931 raft_consensus.cc:374] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.251569 2931 raft_consensus.cc:729] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.251709 2931 consensus_queue.cc:260] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.251948 2931 ts_tablet_manager.cc:1398] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:10.251932 3298 raft_consensus.cc:348] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.252034 3298 raft_consensus.cc:374] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.252065 3298 raft_consensus.cc:729] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.252188 3298 consensus_queue.cc:260] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.252341 3202 tablet_bootstrap.cc:654] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.252534 3298 ts_tablet_manager.cc:1398] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:10.254793 3202 tablet_bootstrap.cc:492] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:10.254881 3202 ts_tablet_manager.cc:1367] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:10.255362 3202 raft_consensus.cc:348] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.255471 3202 raft_consensus.cc:374] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.255503 3202 raft_consensus.cc:729] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.255631 3202 consensus_queue.cc:260] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.255931 3202 ts_tablet_manager.cc:1398] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.266001 24986 consensus_queue.cc:237] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.267594 24594 raft_consensus.cc:1261] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:10.267727 24856 raft_consensus.cc:1261] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:10.267962 3287 consensus_queue.cc:1035] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:10.268131 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 783db67a272147ed8752c5d6dba4456e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:10.268518 3287 consensus_queue.cc:1035] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:10.270323 2502 raft_consensus.cc:2932] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:10.270632 24856 raft_consensus.cc:2932] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:10.272778 24594 raft_consensus.cc:2932] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:10.273659 24447 catalog_manager.cc:5484] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 4, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:10.290318 2149 raft_consensus.cc:482] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:10.290436 2149 raft_consensus.cc:504] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.290815 2149 leader_election.cc:288] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:10.291144 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12749d7e07492c9473fa758c309044" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:10.291306 27032 raft_consensus.cc:2454] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:10.291307 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12749d7e07492c9473fa758c309044" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:10.291424 24986 raft_consensus.cc:2454] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:10.291683 27228 leader_election.cc:302] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:10.291849 2149 raft_consensus.cc:2781] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:10.291893 2149 raft_consensus.cc:482] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:10.291925 2149 raft_consensus.cc:3037] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.292951 2149 raft_consensus.cc:504] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.293277 2149 leader_election.cc:288] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:10.293558 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12749d7e07492c9473fa758c309044" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:10.293665 27032 raft_consensus.cc:3037] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.293716 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12749d7e07492c9473fa758c309044" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:10.293807 24986 raft_consensus.cc:3037] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.294656 27032 raft_consensus.cc:2454] T 0a12749d7e07492c9473fa758c309044 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:10.294762 24986 raft_consensus.cc:2454] T 0a12749d7e07492c9473fa758c309044 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:10.294957 27227 leader_election.cc:302] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:10.295132 2149 raft_consensus.cc:2781] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:10.295223 2149 raft_consensus.cc:686] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:10.295339 2149 consensus_queue.cc:237] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.297236 3183 raft_consensus.cc:482] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:10.297340 3183 raft_consensus.cc:504] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.297808 3183 leader_election.cc:288] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:10.298298 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed5eb90e034c0dbf49880050c0c935" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:10.298475 27162 raft_consensus.cc:2454] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:10.298568 24453 catalog_manager.cc:5484] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:10.298715 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed5eb90e034c0dbf49880050c0c935" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:10.298761 27361 leader_election.cc:302] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:10.298828 24594 raft_consensus.cc:2454] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 0.
I20240501 19:57:10.298934 3183 raft_consensus.cc:2781] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:10.298995 3183 raft_consensus.cc:482] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:10.299052 3183 raft_consensus.cc:3037] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.300067 3183 raft_consensus.cc:504] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.300446 3183 leader_election.cc:288] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:10.300781 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed5eb90e034c0dbf49880050c0c935" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:10.300875 24594 raft_consensus.cc:3037] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.300840 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed5eb90e034c0dbf49880050c0c935" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:10.300980 27162 raft_consensus.cc:3037] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.301769 24594 raft_consensus.cc:2454] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:10.301781 27162 raft_consensus.cc:2454] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6e54e86a9949e5970b42e8e32e7561 in term 1.
I20240501 19:57:10.302060 27361 leader_election.cc:302] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:10.302206 3183 raft_consensus.cc:2781] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:10.302280 3183 raft_consensus.cc:686] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Running, Role: LEADER
I20240501 19:57:10.302387 3183 consensus_queue.cc:237] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.307009 24447 catalog_manager.cc:5484] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8). New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:10.345863 3287 raft_consensus.cc:482] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:10.345990 3287 raft_consensus.cc:504] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.346416 3287 leader_election.cc:288] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:10.346850 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88765fca05c4d31b62db826b51d29c0" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:10.346985 27276 raft_consensus.cc:2454] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:10.346956 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88765fca05c4d31b62db826b51d29c0" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:10.347129 27162 raft_consensus.cc:2454] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:10.347373 24924 leader_election.cc:302] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:10.347644 3287 raft_consensus.cc:2781] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:10.347723 3287 raft_consensus.cc:482] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:10.347762 3287 raft_consensus.cc:3037] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.348757 3287 raft_consensus.cc:504] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.349087 3287 leader_election.cc:288] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:10.350041 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88765fca05c4d31b62db826b51d29c0" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:10.350068 27276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88765fca05c4d31b62db826b51d29c0" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:10.350139 27162 raft_consensus.cc:3037] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.350181 27276 raft_consensus.cc:3037] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:10.351110 27162 raft_consensus.cc:2454] T c88765fca05c4d31b62db826b51d29c0 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:10.351215 27276 raft_consensus.cc:2454] T c88765fca05c4d31b62db826b51d29c0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:10.351425 24924 leader_election.cc:302] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:10.351572 3287 raft_consensus.cc:2781] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:10.351652 3287 raft_consensus.cc:686] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:10.351794 3287 consensus_queue.cc:237] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.357055 24447 catalog_manager.cc:5484] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:10.402676 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:10.416657 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:10.422070 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:10.422574 24836 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.429028 27274 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:10.431025 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.431186 27011 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.436378 27405 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:10.459197 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:10.465378 27162 raft_consensus.cc:1261] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:10.465904 3183 consensus_queue.cc:1035] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:10.467509 24594 raft_consensus.cc:1261] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:10.469404 2517 consensus_queue.cc:1035] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:10.473518 24448 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 9fed5eb90e034c0dbf49880050c0c935 (table rebalancer_dynamic_table [id=6c4db455d9504897b3b366b7178f9d73]). Expected version 1 got 0
I20240501 19:57:10.474085 24447 catalog_manager.cc:5576] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399) does not have the latest schema for tablet 9fed5eb90e034c0dbf49880050c0c935 (table rebalancer_dynamic_table [id=6c4db455d9504897b3b366b7178f9d73]). Expected version 1 got 0
I20240501 19:57:10.474311 3102 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:10.474668 3102 tablet.cc:1620] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:10.477787 24447 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 9fed5eb90e034c0dbf49880050c0c935 (table rebalancer_dynamic_table [id=6c4db455d9504897b3b366b7178f9d73]). Expected version 1 got 0
I20240501 19:57:10.480929 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=6c4db455d9504897b3b366b7178f9d73] alter complete (version 1)
I20240501 19:57:10.491159 3371 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:10.491474 3371 tablet.cc:1620] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:10.494956 3369 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:10.505477 3369 tablet.cc:1620] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:10.596355 24450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:10.603577 3369 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:10.603904 3369 tablet.cc:1620] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:10.605715 3371 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:10.606009 3371 tablet.cc:1620] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:10.609867 3102 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:10.610146 3102 tablet.cc:1620] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:10.612572 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=6c4db455d9504897b3b366b7178f9d73] alter complete (version 2)
I20240501 19:57:10.642475 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.648898 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:10.650012 24836 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.653731 27274 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:10.659622 27011 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.659690 2018 consensus_queue.cc:1035] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:10.661386 3426 ts_tablet_manager.cc:902] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:10.661842 3426 tablet_copy_client.cc:308] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:57:10.662268 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 6ca876a9d51e4096955127e7038e2555 from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:37211)
I20240501 19:57:10.662371 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 6ca876a9d51e4096955127e7038e2555 from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:37211: session id = 9d64cdfaeb2d46b29845160e395fc241-6ca876a9d51e4096955127e7038e2555
I20240501 19:57:10.663314 24616 tablet_copy_source_session.cc:205] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:10.669059 3426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca876a9d51e4096955127e7038e2555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.669646 27405 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:10.672693 27143 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:10.679436 2765 consensus_queue.cc:1035] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:10.690102 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:10.710918 3426 tablet_copy_client.cc:783] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:10.711156 3426 tablet_copy_client.cc:647] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:10.713315 3426 tablet_copy_client.cc:523] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:10.714478 24448 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:10.714826 3426 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:10.719369 3426 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:10.719632 3426 tablet_bootstrap.cc:492] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:57:10.719951 3426 ts_tablet_manager.cc:1367] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 19:57:10.720531 3426 raft_consensus.cc:348] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:10.720664 3426 raft_consensus.cc:729] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.720816 3426 consensus_queue.cc:260] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:10.722345 3426 ts_tablet_manager.cc:1398] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:10.722927 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-6ca876a9d51e4096955127e7038e2555 received from {username='slave'} at 127.17.65.3:37211
I20240501 19:57:10.722931 3369 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:10.723013 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-6ca876a9d51e4096955127e7038e2555 on tablet 6ca876a9d51e4096955127e7038e2555 with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:10.727108 3371 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:10.727767 2149 consensus_queue.cc:1035] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:10.730376 24452 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=6c4db455d9504897b3b366b7178f9d73] alter complete (version 3)
I20240501 19:57:10.733409 2149 consensus_queue.cc:1035] T 0a12749d7e07492c9473fa758c309044 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:10.741690 3102 tablet.cc:1722] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:10.747640 3441 ts_tablet_manager.cc:902] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:10.748414 3441 tablet_copy_client.cc:308] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:57:10.748865 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet dc23e8b5ad8a4911ae766492fb823806 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:57961)
I20240501 19:57:10.748970 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet dc23e8b5ad8a4911ae766492fb823806 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:57961: session id = 4a6e54e86a9949e5970b42e8e32e7561-dc23e8b5ad8a4911ae766492fb823806
I20240501 19:57:10.749722 25009 tablet_copy_source_session.cc:205] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:10.750487 3441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc23e8b5ad8a4911ae766492fb823806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.752732 3441 tablet_copy_client.cc:783] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:10.752988 3441 tablet_copy_client.cc:647] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:10.755056 3441 tablet_copy_client.cc:523] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:10.756357 3441 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:10.783178 3441 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:10.783499 3441 tablet_bootstrap.cc:492] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:57:10.783860 3441 ts_tablet_manager.cc:1367] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.000s
I20240501 19:57:10.784282 3441 raft_consensus.cc:348] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:10.784410 3441 raft_consensus.cc:729] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.784567 3441 consensus_queue.cc:260] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:10.785003 3441 ts_tablet_manager.cc:1398] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.785588 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-dc23e8b5ad8a4911ae766492fb823806 received from {username='slave'} at 127.17.65.8:57961
I20240501 19:57:10.785658 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-dc23e8b5ad8a4911ae766492fb823806 on tablet dc23e8b5ad8a4911ae766492fb823806 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:10.830235 3488 ts_tablet_manager.cc:902] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:10.830760 3488 tablet_copy_client.cc:308] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:57:10.831382 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet 783db67a272147ed8752c5d6dba4456e from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:51823)
I20240501 19:57:10.831506 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet 783db67a272147ed8752c5d6dba4456e from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:51823: session id = 29d1a658e2b946e0b412e5fef82db9d9-783db67a272147ed8752c5d6dba4456e
I20240501 19:57:10.832580 25009 tablet_copy_source_session.cc:205] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:10.835153 3488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783db67a272147ed8752c5d6dba4456e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.837483 3488 tablet_copy_client.cc:783] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:10.837633 3488 tablet_copy_client.cc:647] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:10.839136 3488 tablet_copy_client.cc:523] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:10.840812 3488 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:10.849350 24446 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:10.849480 24446 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:10.851642 24446 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 9fed5eb90e034c0dbf49880050c0c935
I20240501 19:57:10.852463 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fed5eb90e034c0dbf49880050c0c935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:10 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:10.852491 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fed5eb90e034c0dbf49880050c0c935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:10 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:10.853569 3495 tablet_replica.cc:330] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:10.853786 3495 raft_consensus.cc:2229] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:10.854017 3495 raft_consensus.cc:2258] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:10.854699 3495 ts_tablet_manager.cc:1875] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:10.857048 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fed5eb90e034c0dbf49880050c0c935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:10 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:10.860930 3495 ts_tablet_manager.cc:1888] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:10.861047 3495 log.cc:1196] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/9fed5eb90e034c0dbf49880050c0c935
I20240501 19:57:10.861215 3494 tablet_replica.cc:330] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:10.861356 3495 ts_tablet_manager.cc:1909] T 9fed5eb90e034c0dbf49880050c0c935 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:57:10.861425 3494 raft_consensus.cc:2229] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:10.861594 3494 raft_consensus.cc:2258] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:10.862995 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 9fed5eb90e034c0dbf49880050c0c935 (table rebalancer_dynamic_table_new_name [id=6c4db455d9504897b3b366b7178f9d73]) successfully deleted
I20240501 19:57:10.863157 3494 ts_tablet_manager.cc:1875] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:10.864583 3496 tablet_replica.cc:330] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:10.864792 3496 raft_consensus.cc:2229] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:10.864974 3496 raft_consensus.cc:2258] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:10.865536 3494 ts_tablet_manager.cc:1888] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:10.865653 3494 log.cc:1196] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/9fed5eb90e034c0dbf49880050c0c935
I20240501 19:57:10.866869 3494 ts_tablet_manager.cc:1909] T 9fed5eb90e034c0dbf49880050c0c935 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:57:10.867569 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 9fed5eb90e034c0dbf49880050c0c935 (table rebalancer_dynamic_table_new_name [id=6c4db455d9504897b3b366b7178f9d73]) successfully deleted
I20240501 19:57:10.867902 3496 ts_tablet_manager.cc:1875] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:10.871796 3496 ts_tablet_manager.cc:1888] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:10.877972 3488 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:10.878257 3488 tablet_bootstrap.cc:492] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:57:10.878422 3488 ts_tablet_manager.cc:1367] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.008s
I20240501 19:57:10.878908 3488 raft_consensus.cc:348] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.879022 3488 raft_consensus.cc:729] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.879209 3496 log.cc:1196] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/9fed5eb90e034c0dbf49880050c0c935
I20240501 19:57:10.879560 3496 ts_tablet_manager.cc:1909] T 9fed5eb90e034c0dbf49880050c0c935 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:57:10.881270 3488 consensus_queue.cc:260] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.881354 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 9fed5eb90e034c0dbf49880050c0c935 (table rebalancer_dynamic_table_new_name [id=6c4db455d9504897b3b366b7178f9d73]) successfully deleted
I20240501 19:57:10.882551 3488 ts_tablet_manager.cc:1398] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20240501 19:57:10.883167 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-783db67a272147ed8752c5d6dba4456e received from {username='slave'} at 127.17.65.6:51823
I20240501 19:57:10.883253 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-783db67a272147ed8752c5d6dba4456e on tablet 783db67a272147ed8752c5d6dba4456e with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:10.899883 3287 consensus_queue.cc:1035] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:10.904139 3286 consensus_queue.cc:1035] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:10.908779 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:10.916342 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 22f2fc10809948cf82331473f18894b3 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.916394 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 22f2fc10809948cf82331473f18894b3 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.916620 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f2fc10809948cf82331473f18894b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.916620 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f2fc10809948cf82331473f18894b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.917024 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 22f2fc10809948cf82331473f18894b3 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.917311 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f2fc10809948cf82331473f18894b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.918243 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 66b741c6b19f4f1181cdea8a78c816e9 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.918506 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b741c6b19f4f1181cdea8a78c816e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.919220 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 66b741c6b19f4f1181cdea8a78c816e9 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.919450 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b741c6b19f4f1181cdea8a78c816e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.921334 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 518ed5533d024c98b89bb70e9839aaa4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.921600 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518ed5533d024c98b89bb70e9839aaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.922564 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 66b741c6b19f4f1181cdea8a78c816e9 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.922837 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b741c6b19f4f1181cdea8a78c816e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.923134 3511 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:10.924130 3511 tablet_bootstrap.cc:654] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.924640 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 518ed5533d024c98b89bb70e9839aaa4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.924908 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518ed5533d024c98b89bb70e9839aaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.925482 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 518ed5533d024c98b89bb70e9839aaa4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:10.925731 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518ed5533d024c98b89bb70e9839aaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.929087 3512 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:10.929414 3517 tablet_bootstrap.cc:492] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:10.930502 3512 tablet_bootstrap.cc:654] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.931516 3517 tablet_bootstrap.cc:654] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.933447 3511 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:10.933537 3511 ts_tablet_manager.cc:1367] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:57:10.933583 3517 tablet_bootstrap.cc:492] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:10.933666 3517 ts_tablet_manager.cc:1367] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:10.934016 3511 raft_consensus.cc:348] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.934124 3511 raft_consensus.cc:374] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.934165 3511 raft_consensus.cc:729] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.934129 3517 raft_consensus.cc:348] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.934233 3517 raft_consensus.cc:374] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.934316 3517 raft_consensus.cc:729] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.934281 3511 consensus_queue.cc:260] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.934482 3517 consensus_queue.cc:260] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.934657 3511 ts_tablet_manager.cc:1398] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:10.934769 3517 ts_tablet_manager.cc:1398] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.937433 3511 tablet_bootstrap.cc:492] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:10.940958 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:10.944406 3511 tablet_bootstrap.cc:654] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.945179 24836 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.945377 27142 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:10.945480 27011 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.945482 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:10.946455 3511 tablet_bootstrap.cc:492] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:10.946576 3511 ts_tablet_manager.cc:1367] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:10.946995 3511 raft_consensus.cc:348] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.947116 3511 raft_consensus.cc:374] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.947146 3511 raft_consensus.cc:729] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.947247 3511 consensus_queue.cc:260] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.947548 3511 ts_tablet_manager.cc:1398] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.951195 3520 tablet_bootstrap.cc:492] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:10.951414 3515 tablet_bootstrap.cc:492] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:10.951642 3514 tablet_bootstrap.cc:492] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:10.953502 3515 tablet_bootstrap.cc:654] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.954638 3514 tablet_bootstrap.cc:654] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.954752 3520 tablet_bootstrap.cc:654] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.957216 24445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:10.956550 3513 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:10.958734 3513 tablet_bootstrap.cc:654] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.960045 3514 tablet_bootstrap.cc:492] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:10.960170 3514 ts_tablet_manager.cc:1367] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:10.960630 3514 raft_consensus.cc:348] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.960810 3514 raft_consensus.cc:374] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.960883 3514 raft_consensus.cc:729] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.962128 3514 consensus_queue.cc:260] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:10.963915 3514 ts_tablet_manager.cc:1398] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:10.964346 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 093e32291d9d4321acec1a8e5ba2eb03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082]), partition=
I20240501 19:57:10.964592 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093e32291d9d4321acec1a8e5ba2eb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.967252 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 093e32291d9d4321acec1a8e5ba2eb03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082]), partition=
I20240501 19:57:10.967485 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093e32291d9d4321acec1a8e5ba2eb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.967005 27405 tablet_service.cc:1467] Processing CreateTablet for tablet 093e32291d9d4321acec1a8e5ba2eb03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082]), partition=
I20240501 19:57:10.969735 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093e32291d9d4321acec1a8e5ba2eb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:10.972682 3514 tablet_bootstrap.cc:492] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:10.973762 3514 tablet_bootstrap.cc:654] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.974817 3520 tablet_bootstrap.cc:492] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:10.974947 3520 ts_tablet_manager.cc:1367] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 19:57:10.975437 3520 raft_consensus.cc:348] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:10.975638 3520 raft_consensus.cc:374] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.975725 3520 raft_consensus.cc:729] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.975903 3520 consensus_queue.cc:260] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:10.977054 3520 ts_tablet_manager.cc:1398] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:10.977236 3520 tablet_bootstrap.cc:492] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:10.978384 3520 tablet_bootstrap.cc:654] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.980780 3520 tablet_bootstrap.cc:492] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:10.980867 3520 ts_tablet_manager.cc:1367] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:57:10.981204 3520 raft_consensus.cc:348] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:10.981304 3520 raft_consensus.cc:374] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.981338 3520 raft_consensus.cc:729] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.981490 3520 consensus_queue.cc:260] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:10.981827 3520 ts_tablet_manager.cc:1398] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:10.982283 3513 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:10.982411 3513 ts_tablet_manager.cc:1367] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 19:57:10.982786 3513 raft_consensus.cc:348] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.983019 3513 raft_consensus.cc:374] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.983143 3513 raft_consensus.cc:729] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.983358 3513 consensus_queue.cc:260] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:10.983773 3514 tablet_bootstrap.cc:492] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:10.983896 3514 ts_tablet_manager.cc:1367] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:10.984331 3514 raft_consensus.cc:348] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:10.984498 3514 raft_consensus.cc:374] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.984573 3514 raft_consensus.cc:729] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.984732 3514 consensus_queue.cc:260] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:10.985154 3514 ts_tablet_manager.cc:1398] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:10.986325 3513 ts_tablet_manager.cc:1398] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:57:10.986563 3513 tablet_bootstrap.cc:492] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:10.988014 3513 tablet_bootstrap.cc:654] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:10.989082 3513 tablet_bootstrap.cc:492] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:10.989152 3513 ts_tablet_manager.cc:1367] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:10.989528 3513 raft_consensus.cc:348] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:10.989612 3513 raft_consensus.cc:374] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:10.989647 3513 raft_consensus.cc:729] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:10.989761 3513 consensus_queue.cc:260] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:10.990021 3513 ts_tablet_manager.cc:1398] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:10.995658 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:10.995959 27274 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:11.006768 3512 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:11.006889 3512 ts_tablet_manager.cc:1367] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.000s
I20240501 19:57:11.007411 3512 raft_consensus.cc:348] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:11.007541 3512 raft_consensus.cc:374] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.007575 3512 raft_consensus.cc:729] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.007699 3512 consensus_queue.cc:260] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:11.009158 3512 ts_tablet_manager.cc:1398] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:57:11.010056 3515 tablet_bootstrap.cc:492] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:11.010164 3515 ts_tablet_manager.cc:1367] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.002s
I20240501 19:57:11.010656 3515 raft_consensus.cc:348] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:11.010740 3515 raft_consensus.cc:374] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.010761 3515 raft_consensus.cc:729] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.010843 3515 consensus_queue.cc:260] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:11.011303 3515 ts_tablet_manager.cc:1398] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:11.011482 3515 tablet_bootstrap.cc:492] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:11.012296 3515 tablet_bootstrap.cc:654] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.022536 2724 raft_consensus.cc:482] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:11.022648 2724 raft_consensus.cc:504] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:11.023065 2724 leader_election.cc:288] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:11.023447 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f2fc10809948cf82331473f18894b3" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:11.023567 27162 raft_consensus.cc:2454] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:11.024083 24793 leader_election.cc:302] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:11.024679 2724 raft_consensus.cc:2781] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:11.024742 2724 raft_consensus.cc:482] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:11.024785 2724 raft_consensus.cc:3037] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.025856 2724 raft_consensus.cc:504] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:11.026228 2724 leader_election.cc:288] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:11.026624 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f2fc10809948cf82331473f18894b3" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:11.026748 27280 raft_consensus.cc:3037] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.027170 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f2fc10809948cf82331473f18894b3" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:11.028035 27280 raft_consensus.cc:2454] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:11.028307 24793 leader_election.cc:302] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:11.028601 3497 raft_consensus.cc:2781] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:11.028635 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f2fc10809948cf82331473f18894b3" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:11.028699 3497 raft_consensus.cc:686] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:11.028738 27163 raft_consensus.cc:3037] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.028828 3497 consensus_queue.cc:237] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:11.029840 27163 raft_consensus.cc:2454] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:11.030966 24452 catalog_manager.cc:5484] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:11.037835 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:11.040866 3515 tablet_bootstrap.cc:492] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:11.040966 3515 ts_tablet_manager.cc:1367] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.000s
I20240501 19:57:11.041384 3515 raft_consensus.cc:348] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.041502 3515 raft_consensus.cc:374] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.041540 3515 raft_consensus.cc:729] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.041658 3515 consensus_queue.cc:260] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.041960 3515 ts_tablet_manager.cc:1398] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:57:11.082671 27163 consensus_queue.cc:237] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:11.083878 24594 raft_consensus.cc:1261] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:11.083878 27423 raft_consensus.cc:1261] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:11.084228 2437 consensus_queue.cc:1035] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:11.084393 3424 consensus_queue.cc:1035] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:11.086081 3424 raft_consensus.cc:2932] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:11.086473 27423 raft_consensus.cc:2932] T 8b747487a083433d91191aa79458f51e P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:11.086910 24594 raft_consensus.cc:2932] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:11.089812 24448 catalog_manager.cc:5484] T 8b747487a083433d91191aa79458f51e P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:11.093814 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b747487a083433d91191aa79458f51e with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:11.094271 3553 tablet_replica.cc:330] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:11.094542 3553 raft_consensus.cc:2229] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:11.094705 3553 raft_consensus.cc:2258] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:11.095194 3553 ts_tablet_manager.cc:1875] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:11.097735 3553 ts_tablet_manager.cc:1888] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:11.097813 3553 log.cc:1196] T 8b747487a083433d91191aa79458f51e P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/8b747487a083433d91191aa79458f51e
I20240501 19:57:11.098500 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 8b747487a083433d91191aa79458f51e (table rebalancer_extra_table_10 [id=6b4f6cd45ea3442a81b44213484ac86f]) successfully deleted
I20240501 19:57:11.147742 3431 raft_consensus.cc:482] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:11.147876 3431 raft_consensus.cc:504] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:11.148365 3431 leader_election.cc:288] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:11.148813 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518ed5533d024c98b89bb70e9839aaa4" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:11.148941 24856 raft_consensus.cc:2454] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:11.149497 26966 leader_election.cc:302] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:11.149825 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518ed5533d024c98b89bb70e9839aaa4" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:11.149889 27423 raft_consensus.cc:1203] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:11.149956 24725 raft_consensus.cc:2454] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:11.151934 3431 raft_consensus.cc:2781] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:11.151995 3431 raft_consensus.cc:482] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:11.152029 3431 raft_consensus.cc:3037] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.153173 3431 raft_consensus.cc:504] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:11.153487 3431 leader_election.cc:288] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:11.154867 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518ed5533d024c98b89bb70e9839aaa4" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:11.154971 24856 raft_consensus.cc:3037] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.156004 24856 raft_consensus.cc:2454] T 518ed5533d024c98b89bb70e9839aaa4 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:11.156577 26966 leader_election.cc:302] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:11.157030 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518ed5533d024c98b89bb70e9839aaa4" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:11.157150 24725 raft_consensus.cc:3037] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.158098 24725 raft_consensus.cc:2454] T 518ed5533d024c98b89bb70e9839aaa4 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:11.158269 3424 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:11.158375 3424 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:11.158725 3424 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:11.158883 2018 raft_consensus.cc:2781] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:11.158969 2018 raft_consensus.cc:686] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:11.159121 2018 consensus_queue.cc:237] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:11.159221 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b741c6b19f4f1181cdea8a78c816e9" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:11.159332 24986 raft_consensus.cc:2454] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:11.159601 27097 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:11.159751 3424 raft_consensus.cc:2781] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:11.159802 3424 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:11.159834 3424 raft_consensus.cc:3037] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.160405 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b741c6b19f4f1181cdea8a78c816e9" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:11.160532 27423 raft_consensus.cc:2454] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:11.160732 3424 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:11.161054 3424 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:11.161885 24447 catalog_manager.cc:5484] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:11.161996 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b741c6b19f4f1181cdea8a78c816e9" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:11.162092 27423 raft_consensus.cc:3037] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.163017 27423 raft_consensus.cc:2454] T 66b741c6b19f4f1181cdea8a78c816e9 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:11.163288 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b741c6b19f4f1181cdea8a78c816e9" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:11.163388 24986 raft_consensus.cc:3037] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.163331 27098 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:11.163516 3424 raft_consensus.cc:2781] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:11.163594 3424 raft_consensus.cc:686] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:11.163713 3424 consensus_queue.cc:237] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:11.164323 24986 raft_consensus.cc:2454] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:11.167393 24447 catalog_manager.cc:5484] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:11.182984 24856 raft_consensus.cc:1203] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:11.233063 3431 raft_consensus.cc:482] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:11.233196 3431 raft_consensus.cc:504] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.233577 3431 leader_election.cc:288] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:11.233880 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093e32291d9d4321acec1a8e5ba2eb03" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:11.233909 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093e32291d9d4321acec1a8e5ba2eb03" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:11.234007 24725 raft_consensus.cc:2454] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:11.234033 27423 raft_consensus.cc:2454] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:11.234308 26965 leader_election.cc:302] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:11.234462 3431 raft_consensus.cc:2781] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:11.234508 3431 raft_consensus.cc:482] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:11.234547 3431 raft_consensus.cc:3037] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.235600 3431 raft_consensus.cc:504] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.235935 3431 leader_election.cc:288] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:11.236220 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093e32291d9d4321acec1a8e5ba2eb03" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:11.236351 27423 raft_consensus.cc:3037] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.236342 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093e32291d9d4321acec1a8e5ba2eb03" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:11.236469 24725 raft_consensus.cc:3037] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.237346 27423 raft_consensus.cc:2454] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:11.237437 24725 raft_consensus.cc:2454] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:11.237627 26967 leader_election.cc:302] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:11.237797 3431 raft_consensus.cc:2781] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:11.237900 3431 raft_consensus.cc:686] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:11.238031 3431 consensus_queue.cc:237] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.240509 24445 catalog_manager.cc:5484] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:11.256376 27142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:11.264782 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:11.265669 27011 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:11.274899 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:11.283212 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:11.292755 24836 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:11.299198 27274 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:11.302040 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:11.359882 24856 consensus_queue.cc:237] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:11.360998 24594 raft_consensus.cc:1261] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:11.361037 27162 raft_consensus.cc:1261] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:11.361443 2724 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:11.361817 3497 consensus_queue.cc:1035] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:11.363312 2724 raft_consensus.cc:2932] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:11.363761 24594 raft_consensus.cc:2932] T d29130c48f7a435498e854973a090b37 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:11.367309 24445 catalog_manager.cc:5484] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 2 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New cstate: current_term: 2 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:11.370115 27163 raft_consensus.cc:2932] T d29130c48f7a435498e854973a090b37 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:11.371309 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet d29130c48f7a435498e854973a090b37 with delete_type TABLET_DATA_TOMBSTONED (TS 3b5968cebd0b4294898fdd89b7a5b3a1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:11.371695 3647 tablet_replica.cc:330] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:11.371881 3647 raft_consensus.cc:2229] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:11.371997 3647 raft_consensus.cc:2258] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:11.372495 3647 ts_tablet_manager.cc:1875] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:11.374819 3647 ts_tablet_manager.cc:1888] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 19:57:11.374927 3647 log.cc:1196] T d29130c48f7a435498e854973a090b37 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/d29130c48f7a435498e854973a090b37
I20240501 19:57:11.375648 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet d29130c48f7a435498e854973a090b37 (table rebalancer_extra_table_12 [id=d3f59491f47f42698971cb82f7aa32d3]) successfully deleted
I20240501 19:57:11.390082 24449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:11.394661 27163 raft_consensus.cc:1203] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:11.396065 24725 raft_consensus.cc:1261] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:11.396169 27423 raft_consensus.cc:1261] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:11.396889 3431 consensus_queue.cc:1035] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:11.397020 2018 consensus_queue.cc:1035] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:11.402294 24445 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 093e32291d9d4321acec1a8e5ba2eb03 (table rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082]). Expected version 1 got 0
I20240501 19:57:11.404312 3664 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:11.404639 3664 tablet.cc:1620] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:11.409277 24445 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 093e32291d9d4321acec1a8e5ba2eb03 (table rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082]). Expected version 1 got 0
I20240501 19:57:11.411193 24449 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082] alter complete (version 1)
I20240501 19:57:11.410856 3665 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:11.412890 24445 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 093e32291d9d4321acec1a8e5ba2eb03 (table rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082]). Expected version 1 got 0
I20240501 19:57:11.414599 24445 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 093e32291d9d4321acec1a8e5ba2eb03 (table rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082]). Expected version 1 got 0
I20240501 19:57:11.417992 3665 tablet.cc:1620] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:11.419180 3667 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:11.419518 3667 tablet.cc:1620] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:11.473959 2724 consensus_queue.cc:1035] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:11.478977 3000 consensus_queue.cc:1035] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:11.516556 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:11.529541 3664 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:11.529845 3664 tablet.cc:1620] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:11.533787 3667 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:11.534142 3667 tablet.cc:1620] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:11.535938 3665 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:11.536223 3665 tablet.cc:1620] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:11.540616 24451 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=2d35a32c73d14e1dbf4a5295b8d55082] alter complete (version 2)
I20240501 19:57:11.541731 24836 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:11.542065 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:11.543763 24705 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:11.550182 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:11.551375 3431 consensus_queue.cc:1035] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:11.552256 27274 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:11.558193 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:11.559217 27011 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:11.561259 27142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:11.561563 2018 consensus_queue.cc:1035] T 518ed5533d024c98b89bb70e9839aaa4 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:11.586567 3424 consensus_queue.cc:1035] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:11.633245 2437 consensus_queue.cc:1035] T 66b741c6b19f4f1181cdea8a78c816e9 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:11.650568 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:11.655838 3664 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:11.656508 3667 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:11.656565 3665 tablet.cc:1722] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:11.661839 24450 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=2d35a32c73d14e1dbf4a5295b8d55082] alter complete (version 3)
I20240501 19:57:11.672029 24986 consensus_queue.cc:237] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:11.673141 27424 raft_consensus.cc:1261] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:11.673179 24725 raft_consensus.cc:1261] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:11.673530 3286 consensus_queue.cc:1035] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:11.673715 3287 consensus_queue.cc:1035] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:11.675227 3286 raft_consensus.cc:2932] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:11.675796 27424 raft_consensus.cc:2932] T dc23e8b5ad8a4911ae766492fb823806 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:11.677089 24725 raft_consensus.cc:2932] T dc23e8b5ad8a4911ae766492fb823806 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:11.677901 24450 catalog_manager.cc:5484] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:11.680512 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet dc23e8b5ad8a4911ae766492fb823806 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:11.680877 3738 tablet_replica.cc:330] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:11.681043 3738 raft_consensus.cc:2229] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:11.681247 3738 raft_consensus.cc:2258] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:11.681821 3738 ts_tablet_manager.cc:1875] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:11.684283 3738 ts_tablet_manager.cc:1888] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:11.684360 3738 log.cc:1196] T dc23e8b5ad8a4911ae766492fb823806 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc23e8b5ad8a4911ae766492fb823806
I20240501 19:57:11.684921 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet dc23e8b5ad8a4911ae766492fb823806 (table rebalance_test_table_7 [id=deae6feaf63e416882a15a456246b972]) successfully deleted
I20240501 19:57:11.703462 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:11.705063 24594 consensus_queue.cc:237] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.706476 24856 raft_consensus.cc:1261] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:11.706826 24529 consensus_peers.cc:485] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3d31791486b4e68a9e5283faadb3f8c. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:11.707134 2539 consensus_queue.cc:1035] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:11.707267 24986 raft_consensus.cc:1261] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:11.707764 2539 consensus_queue.cc:1035] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:11.710372 2539 raft_consensus.cc:2932] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:11.710733 24986 raft_consensus.cc:2932] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:11.713775 24451 catalog_manager.cc:5484] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:11.714804 24966 tablet_service.cc:1467] Processing CreateTablet for tablet f76202780c824bc0b3debfe8728fc007 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.715132 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76202780c824bc0b3debfe8728fc007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.715246 24836 tablet_service.cc:1467] Processing CreateTablet for tablet f76202780c824bc0b3debfe8728fc007 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.715513 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76202780c824bc0b3debfe8728fc007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.715812 27405 tablet_service.cc:1467] Processing CreateTablet for tablet f76202780c824bc0b3debfe8728fc007 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.715864 27404 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.716050 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76202780c824bc0b3debfe8728fc007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.716572 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.717742 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.717979 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.718300 3766 tablet_bootstrap.cc:492] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:11.718461 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 4b48df4b84f24a7fa514f802a5442445 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.718665 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b48df4b84f24a7fa514f802a5442445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.719291 3766 tablet_bootstrap.cc:654] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.720335 3766 tablet_bootstrap.cc:492] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:11.720413 3766 ts_tablet_manager.cc:1367] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:11.720606 24856 raft_consensus.cc:2932] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:11.720877 3766 raft_consensus.cc:348] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.720990 3766 raft_consensus.cc:374] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.721024 3766 raft_consensus.cc:729] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.721136 3766 consensus_queue.cc:260] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.721424 3766 ts_tablet_manager.cc:1398] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:11.721604 3766 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:11.722545 3766 tablet_bootstrap.cc:654] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.723351 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.723618 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.723634 27012 tablet_service.cc:1467] Processing CreateTablet for tablet 4b48df4b84f24a7fa514f802a5442445 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.723851 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b48df4b84f24a7fa514f802a5442445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.724205 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 4b48df4b84f24a7fa514f802a5442445 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:11.724452 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b48df4b84f24a7fa514f802a5442445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.725764 3770 tablet_bootstrap.cc:492] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:11.726253 3771 tablet_bootstrap.cc:492] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:11.727007 3770 tablet_bootstrap.cc:654] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.727517 3771 tablet_bootstrap.cc:654] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.728044 3770 tablet_bootstrap.cc:492] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:11.728111 3770 ts_tablet_manager.cc:1367] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:11.728540 3770 raft_consensus.cc:348] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.728626 3770 raft_consensus.cc:374] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.728655 3770 raft_consensus.cc:729] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.728781 3770 consensus_queue.cc:260] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.729143 3770 ts_tablet_manager.cc:1398] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:11.733727 3770 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:11.734253 3764 tablet_bootstrap.cc:492] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:11.734454 3771 tablet_bootstrap.cc:492] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:11.734485 3768 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:11.734555 3771 ts_tablet_manager.cc:1367] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:57:11.734890 3770 tablet_bootstrap.cc:654] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.734997 3771 raft_consensus.cc:348] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.735111 3771 raft_consensus.cc:374] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.735144 3771 raft_consensus.cc:729] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.735257 3771 consensus_queue.cc:260] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.735567 3771 ts_tablet_manager.cc:1398] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:11.736541 3766 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:11.736625 3766 ts_tablet_manager.cc:1367] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240501 19:57:11.737087 3766 raft_consensus.cc:348] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.737207 3766 raft_consensus.cc:374] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.737239 3766 raft_consensus.cc:729] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.737367 3766 consensus_queue.cc:260] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.737462 3770 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:11.737533 3770 ts_tablet_manager.cc:1367] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:57:11.737653 3766 ts_tablet_manager.cc:1398] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:11.737887 3766 tablet_bootstrap.cc:492] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:11.737969 3770 raft_consensus.cc:348] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.738073 3770 raft_consensus.cc:374] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.738106 3770 raft_consensus.cc:729] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.738243 3770 consensus_queue.cc:260] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.738504 3770 ts_tablet_manager.cc:1398] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:11.738941 3766 tablet_bootstrap.cc:654] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.739395 3764 tablet_bootstrap.cc:654] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.738950 3768 tablet_bootstrap.cc:654] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.740557 3766 tablet_bootstrap.cc:492] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:11.740700 3766 ts_tablet_manager.cc:1367] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:11.740801 3765 tablet_bootstrap.cc:492] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:11.741233 3766 raft_consensus.cc:348] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.741407 3766 raft_consensus.cc:374] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.741482 3766 raft_consensus.cc:729] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.741681 3766 consensus_queue.cc:260] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.741815 3765 tablet_bootstrap.cc:654] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.742154 3766 ts_tablet_manager.cc:1398] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:11.743492 3768 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:11.743628 3768 ts_tablet_manager.cc:1367] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:11.745501 3768 raft_consensus.cc:348] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.745576 3765 tablet_bootstrap.cc:492] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:11.745605 3768 raft_consensus.cc:374] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.745653 3765 ts_tablet_manager.cc:1367] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:11.745673 3768 raft_consensus.cc:729] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.745942 3768 consensus_queue.cc:260] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.746260 3768 ts_tablet_manager.cc:1398] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:11.748608 3765 raft_consensus.cc:348] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.749023 3765 raft_consensus.cc:374] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.749114 3765 raft_consensus.cc:729] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.749269 3765 consensus_queue.cc:260] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.750459 3765 ts_tablet_manager.cc:1398] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:11.759179 3764 tablet_bootstrap.cc:492] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:11.759361 3764 ts_tablet_manager.cc:1367] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20240501 19:57:11.759914 3764 raft_consensus.cc:348] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.760074 3764 raft_consensus.cc:374] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.760130 3764 raft_consensus.cc:729] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.760277 3764 consensus_queue.cc:260] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.760707 3764 ts_tablet_manager.cc:1398] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:11.764246 24450 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:11.764397 24450 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:11.766796 24450 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 093e32291d9d4321acec1a8e5ba2eb03
I20240501 19:57:11.767450 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 093e32291d9d4321acec1a8e5ba2eb03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:11 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:11.768250 24594 consensus_queue.cc:237] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:11.769170 27011 tablet_service.cc:1514] Processing DeleteTablet for tablet 093e32291d9d4321acec1a8e5ba2eb03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:11 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:11.769351 3647 tablet_replica.cc:330] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:11.769507 3647 raft_consensus.cc:2229] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:11.769768 3647 raft_consensus.cc:2258] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 19:57:11.769832 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7bc7629944b3da3fe8fb1ecae32b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:11.770094 24725 raft_consensus.cc:1261] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:11.770529 3760 consensus_queue.cc:1035] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:11.771096 27406 tablet_service.cc:1514] Processing DeleteTablet for tablet 093e32291d9d4321acec1a8e5ba2eb03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:11 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:11.772091 24856 raft_consensus.cc:1261] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:11.772444 3760 consensus_queue.cc:1035] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:11.774339 3647 ts_tablet_manager.cc:1875] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:11.775511 3796 tablet_replica.cc:330] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:11.775702 3796 raft_consensus.cc:2229] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:11.775908 3796 raft_consensus.cc:2258] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:11.776718 3647 ts_tablet_manager.cc:1888] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:11.776798 3647 log.cc:1196] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/093e32291d9d4321acec1a8e5ba2eb03
I20240501 19:57:11.777304 3796 ts_tablet_manager.cc:1875] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:11.777547 3647 ts_tablet_manager.cc:1909] T 093e32291d9d4321acec1a8e5ba2eb03 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:11.778323 3760 raft_consensus.cc:2932] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:11.781276 24725 raft_consensus.cc:2932] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:11.781764 3796 ts_tablet_manager.cc:1888] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:11.781826 3796 log.cc:1196] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/093e32291d9d4321acec1a8e5ba2eb03
I20240501 19:57:11.782099 3796 ts_tablet_manager.cc:1909] T 093e32291d9d4321acec1a8e5ba2eb03 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:57:11.783490 24450 catalog_manager.cc:5484] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:11.784119 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 093e32291d9d4321acec1a8e5ba2eb03 (table rebalancer_dynamic_table_new_name [id=2d35a32c73d14e1dbf4a5295b8d55082]) successfully deleted
I20240501 19:57:11.784046 24856 raft_consensus.cc:2932] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:11.785135 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 093e32291d9d4321acec1a8e5ba2eb03 (table rebalancer_dynamic_table_new_name [id=2d35a32c73d14e1dbf4a5295b8d55082]) successfully deleted
I20240501 19:57:11.785847 3794 tablet_replica.cc:330] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:11.786020 3794 raft_consensus.cc:2229] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:11.786216 3794 raft_consensus.cc:2258] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:11.787807 3794 ts_tablet_manager.cc:1875] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:11.793260 3794 ts_tablet_manager.cc:1888] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:11.793391 3794 log.cc:1196] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/093e32291d9d4321acec1a8e5ba2eb03
I20240501 19:57:11.793763 3794 ts_tablet_manager.cc:1909] T 093e32291d9d4321acec1a8e5ba2eb03 P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:57:11.794391 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 093e32291d9d4321acec1a8e5ba2eb03 (table rebalancer_dynamic_table_new_name [id=2d35a32c73d14e1dbf4a5295b8d55082]) successfully deleted
I20240501 19:57:11.842725 3440 raft_consensus.cc:482] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:11.842837 3440 raft_consensus.cc:504] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.843209 3440 leader_election.cc:288] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:11.843514 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3a2147d24e4fbf92aa1eb88bfb4881" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:11.843551 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3a2147d24e4fbf92aa1eb88bfb4881" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:11.843657 27423 raft_consensus.cc:2454] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:11.843653 27032 raft_consensus.cc:2454] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 0.
I20240501 19:57:11.843905 24660 leader_election.cc:302] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:11.844172 3440 raft_consensus.cc:2781] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:11.844233 3440 raft_consensus.cc:482] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:11.844280 3440 raft_consensus.cc:3037] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.845621 3440 raft_consensus.cc:504] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.845970 3440 leader_election.cc:288] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:11.846372 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3a2147d24e4fbf92aa1eb88bfb4881" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:11.846448 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3a2147d24e4fbf92aa1eb88bfb4881" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:11.846482 27423 raft_consensus.cc:3037] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.846554 27032 raft_consensus.cc:3037] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.847527 27423 raft_consensus.cc:2454] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:11.847527 27032 raft_consensus.cc:2454] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:11.847993 24660 leader_election.cc:302] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:11.848176 3440 raft_consensus.cc:2781] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:11.848327 3440 raft_consensus.cc:686] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:11.848465 3440 consensus_queue.cc:237] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:11.854090 24451 catalog_manager.cc:5484] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:11.870375 3497 raft_consensus.cc:482] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:11.870484 3497 raft_consensus.cc:504] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.870893 3497 leader_election.cc:288] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:11.871204 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76202780c824bc0b3debfe8728fc007" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:11.871327 27423 raft_consensus.cc:2454] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:11.871568 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76202780c824bc0b3debfe8728fc007" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:11.871711 24986 raft_consensus.cc:2454] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 0.
I20240501 19:57:11.872300 24792 leader_election.cc:302] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:11.872498 24453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:11.872571 3497 raft_consensus.cc:2781] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:11.872629 3497 raft_consensus.cc:482] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:11.872666 3497 raft_consensus.cc:3037] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.873749 3497 raft_consensus.cc:504] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.874095 3497 leader_election.cc:288] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:11.874751 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76202780c824bc0b3debfe8728fc007" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:11.874825 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76202780c824bc0b3debfe8728fc007" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:11.874866 24986 raft_consensus.cc:3037] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.874927 27423 raft_consensus.cc:3037] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.876053 27423 raft_consensus.cc:2454] T f76202780c824bc0b3debfe8728fc007 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:11.876051 24986 raft_consensus.cc:2454] T f76202780c824bc0b3debfe8728fc007 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 1.
I20240501 19:57:11.876328 24793 leader_election.cc:302] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:11.876523 3497 raft_consensus.cc:2781] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:11.876595 3497 raft_consensus.cc:686] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:11.876754 3497 consensus_queue.cc:237] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.877825 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 4c812d2297ad4affa36bdfa865b9498b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b]), partition=
I20240501 19:57:11.878062 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c812d2297ad4affa36bdfa865b9498b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.878782 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 4c812d2297ad4affa36bdfa865b9498b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b]), partition=
I20240501 19:57:11.879068 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c812d2297ad4affa36bdfa865b9498b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.880421 3843 tablet_bootstrap.cc:492] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:11.881300 3843 tablet_bootstrap.cc:654] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.881525 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 4c812d2297ad4affa36bdfa865b9498b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b]), partition=
I20240501 19:57:11.881740 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c812d2297ad4affa36bdfa865b9498b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:11.885587 3765 tablet_bootstrap.cc:492] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:11.885855 24454 catalog_manager.cc:5484] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:11.886346 3765 tablet_bootstrap.cc:654] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.886917 3844 tablet_bootstrap.cc:492] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:11.891007 3844 tablet_bootstrap.cc:654] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:11.892186 3843 tablet_bootstrap.cc:492] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:11.892251 3844 tablet_bootstrap.cc:492] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:11.892328 3843 ts_tablet_manager.cc:1367] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:11.892378 3844 ts_tablet_manager.cc:1367] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 19:57:11.892844 3843 raft_consensus.cc:348] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.892983 3843 raft_consensus.cc:374] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.892948 3844 raft_consensus.cc:348] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.893029 3843 raft_consensus.cc:729] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.893078 3844 raft_consensus.cc:374] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.893124 3844 raft_consensus.cc:729] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.893160 3843 consensus_queue.cc:260] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.893247 3844 consensus_queue.cc:260] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.893545 3844 ts_tablet_manager.cc:1398] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:11.894542 3843 ts_tablet_manager.cc:1398] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:11.897847 3765 tablet_bootstrap.cc:492] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:11.897969 3765 ts_tablet_manager.cc:1367] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:11.898522 3765 raft_consensus.cc:348] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.898619 3765 raft_consensus.cc:374] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:11.898653 3765 raft_consensus.cc:729] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:11.898761 3765 consensus_queue.cc:260] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.899072 3765 ts_tablet_manager.cc:1398] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:11.954223 1038 raft_consensus.cc:482] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:11.954334 1038 raft_consensus.cc:504] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.954689 1038 leader_election.cc:288] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:11.954936 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c812d2297ad4affa36bdfa865b9498b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:11.954995 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c812d2297ad4affa36bdfa865b9498b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:11.955092 27163 raft_consensus.cc:2454] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:11.955148 24857 raft_consensus.cc:2454] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:11.955431 27228 leader_election.cc:302] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:11.955602 1038 raft_consensus.cc:2781] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:11.955691 1038 raft_consensus.cc:482] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:11.955739 1038 raft_consensus.cc:3037] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.956568 1038 raft_consensus.cc:504] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.956902 1038 leader_election.cc:288] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:11.957149 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c812d2297ad4affa36bdfa865b9498b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:11.957165 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c812d2297ad4affa36bdfa865b9498b" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:11.957243 27163 raft_consensus.cc:3037] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.957276 24857 raft_consensus.cc:3037] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:11.957970 27163 raft_consensus.cc:2454] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:11.958297 24857 raft_consensus.cc:2454] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:11.958307 27229 leader_election.cc:302] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:11.958457 2149 raft_consensus.cc:2781] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:11.958600 2149 raft_consensus.cc:686] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:11.958727 2149 consensus_queue.cc:237] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:11.961004 24453 catalog_manager.cc:5484] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:12.050763 3431 raft_consensus.cc:482] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:12.050866 3431 raft_consensus.cc:504] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:12.051257 3431 leader_election.cc:288] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:12.051548 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b48df4b84f24a7fa514f802a5442445" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:12.051687 27423 raft_consensus.cc:2454] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:12.051788 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b48df4b84f24a7fa514f802a5442445" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:12.051913 24594 raft_consensus.cc:2454] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:12.052354 26965 leader_election.cc:302] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:12.052560 3714 raft_consensus.cc:2781] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:12.052645 3714 raft_consensus.cc:482] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:12.052693 3714 raft_consensus.cc:3037] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.053875 3714 raft_consensus.cc:504] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:12.054240 3714 leader_election.cc:288] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:12.054525 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b48df4b84f24a7fa514f802a5442445" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:12.054687 27423 raft_consensus.cc:3037] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.054699 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b48df4b84f24a7fa514f802a5442445" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:12.054809 24594 raft_consensus.cc:3037] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.055792 24594 raft_consensus.cc:2454] T 4b48df4b84f24a7fa514f802a5442445 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:12.055915 27423 raft_consensus.cc:2454] T 4b48df4b84f24a7fa514f802a5442445 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:12.056159 26967 leader_election.cc:302] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:12.056488 3714 raft_consensus.cc:2781] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:12.056591 3714 raft_consensus.cc:686] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:12.056754 3714 consensus_queue.cc:237] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:12.058923 24453 catalog_manager.cc:5484] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:12.068527 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:12.073724 24857 raft_consensus.cc:1261] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:12.074066 27163 raft_consensus.cc:1261] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:12.074908 3848 consensus_queue.cc:1035] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.076378 1038 consensus_queue.cc:1035] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.079938 24453 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 4c812d2297ad4affa36bdfa865b9498b (table rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b]). Expected version 1 got 0
I20240501 19:57:12.083382 27011 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:12.087304 24453 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet 4c812d2297ad4affa36bdfa865b9498b (table rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b]). Expected version 1 got 0
I20240501 19:57:12.087782 27142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:12.088045 27406 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:12.088968 3871 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:12.089356 3871 tablet.cc:1620] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:12.092315 24453 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet 4c812d2297ad4affa36bdfa865b9498b (table rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b]). Expected version 1 got 0
I20240501 19:57:12.093241 24450 catalog_manager.cc:5576] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579) does not have the latest schema for tablet 4c812d2297ad4affa36bdfa865b9498b (table rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b]). Expected version 1 got 0
I20240501 19:57:12.096760 3873 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:12.097152 3873 tablet.cc:1620] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:12.101071 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b] alter complete (version 1)
I20240501 19:57:12.102157 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:12.105289 3872 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:12.106637 3872 tablet.cc:1620] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:12.112247 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:12.129866 3874 ts_tablet_manager.cc:902] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:12.135433 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:12.136232 3874 tablet_copy_client.cc:308] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:57:12.136274 27274 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:12.140769 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet c3d31791486b4e68a9e5283faadb3f8c from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:41295)
I20240501 19:57:12.140859 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet c3d31791486b4e68a9e5283faadb3f8c from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:41295: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-c3d31791486b4e68a9e5283faadb3f8c
I20240501 19:57:12.141710 24616 tablet_copy_source_session.cc:205] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:12.142681 3874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d31791486b4e68a9e5283faadb3f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.143188 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:12.147953 3874 tablet_copy_client.cc:783] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:12.148119 3874 tablet_copy_client.cc:647] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:12.149040 3874 tablet_copy_client.cc:523] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:12.151199 3874 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:12.199195 3874 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:12.199527 3874 tablet_bootstrap.cc:492] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:57:12.199743 3874 ts_tablet_manager.cc:1367] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.049s user 0.022s sys 0.005s
I20240501 19:57:12.200233 3874 raft_consensus.cc:348] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:12.200363 3874 raft_consensus.cc:729] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.200529 3874 consensus_queue.cc:260] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:12.200927 3874 ts_tablet_manager.cc:1398] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:12.201629 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-c3d31791486b4e68a9e5283faadb3f8c received from {username='slave'} at 127.17.65.5:41295
I20240501 19:57:12.201706 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-c3d31791486b4e68a9e5283faadb3f8c on tablet c3d31791486b4e68a9e5283faadb3f8c with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:57:12.207511 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:12.213404 3871 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:12.213703 3871 tablet.cc:1620] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:12.214033 3872 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:12.214313 3872 tablet.cc:1620] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:12.214603 3873 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:12.214856 3873 tablet.cc:1620] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:12.219957 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=5943530919264671886a7a554bf2636b] alter complete (version 2)
I20240501 19:57:12.305836 3440 consensus_queue.cc:1035] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.308694 3440 consensus_queue.cc:1035] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.323873 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:12.328807 3497 consensus_queue.cc:1035] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.329494 27142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:12.334252 3000 consensus_queue.cc:1035] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.341351 27274 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:12.342780 3871 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:12.343183 3872 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:12.343322 24574 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:12.343750 27406 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:12.346762 3873 tablet.cc:1722] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:12.353437 27011 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:12.356639 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=5943530919264671886a7a554bf2636b] alter complete (version 3)
I20240501 19:57:12.367122 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:12.374326 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:12.381088 24705 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:12.396934 3940 ts_tablet_manager.cc:902] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:12.397583 3940 tablet_copy_client.cc:308] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:57:12.398123 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 39e7bc7629944b3da3fe8fb1ecae32b1 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:56477)
I20240501 19:57:12.398218 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 39e7bc7629944b3da3fe8fb1ecae32b1 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:56477: session id = 4a6e54e86a9949e5970b42e8e32e7561-39e7bc7629944b3da3fe8fb1ecae32b1
I20240501 19:57:12.399214 24616 tablet_copy_source_session.cc:205] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:12.402077 3940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e7bc7629944b3da3fe8fb1ecae32b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.405057 3940 tablet_copy_client.cc:783] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:12.405268 3940 tablet_copy_client.cc:647] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:12.406383 3940 tablet_copy_client.cc:523] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:12.407539 3940 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:12.434278 3940 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:12.434638 3940 tablet_bootstrap.cc:492] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:57:12.434837 3940 ts_tablet_manager.cc:1367] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.003s
I20240501 19:57:12.435261 3940 raft_consensus.cc:348] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:12.435377 3940 raft_consensus.cc:729] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.435518 3940 consensus_queue.cc:260] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:12.437031 3940 ts_tablet_manager.cc:1398] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 19:57:12.437916 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-39e7bc7629944b3da3fe8fb1ecae32b1 received from {username='slave'} at 127.17.65.8:56477
I20240501 19:57:12.438002 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-39e7bc7629944b3da3fe8fb1ecae32b1 on tablet 39e7bc7629944b3da3fe8fb1ecae32b1 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:12.461315 24986 consensus_queue.cc:237] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:12.462375 27163 raft_consensus.cc:1261] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:12.462456 24594 raft_consensus.cc:1261] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:12.462764 3286 consensus_queue.cc:1035] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:12.462924 3623 consensus_queue.cc:1035] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:12.464704 3964 raft_consensus.cc:2932] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:12.464890 27163 raft_consensus.cc:2932] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:12.464954 24594 raft_consensus.cc:2932] T 783db67a272147ed8752c5d6dba4456e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:12.468016 24450 catalog_manager.cc:5484] T 783db67a272147ed8752c5d6dba4456e P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:12.468659 24454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:12.468763 24454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:12.470924 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 783db67a272147ed8752c5d6dba4456e with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:12.471695 24454 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 4c812d2297ad4affa36bdfa865b9498b
I20240501 19:57:12.472178 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c812d2297ad4affa36bdfa865b9498b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:12 UTC) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:12.472654 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c812d2297ad4affa36bdfa865b9498b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:12 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:12.472698 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c812d2297ad4affa36bdfa865b9498b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:12 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:12.473192 3967 tablet_replica.cc:330] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:12.473459 3967 raft_consensus.cc:2229] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:12.473714 3967 raft_consensus.cc:2258] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:12.474320 3965 tablet_replica.cc:330] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:12.474331 3967 ts_tablet_manager.cc:1875] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:12.474557 3965 raft_consensus.cc:2229] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:12.474756 3965 raft_consensus.cc:2258] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:12.475303 3965 ts_tablet_manager.cc:1875] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:12.476521 3967 ts_tablet_manager.cc:1888] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:12.476644 3967 log.cc:1196] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/4c812d2297ad4affa36bdfa865b9498b
I20240501 19:57:12.477016 3967 ts_tablet_manager.cc:1909] T 4c812d2297ad4affa36bdfa865b9498b P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:57:12.477730 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 4c812d2297ad4affa36bdfa865b9498b (table rebalancer_dynamic_table_new_name [id=5943530919264671886a7a554bf2636b]) successfully deleted
I20240501 19:57:12.477952 3966 tablet_replica.cc:330] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:12.478120 3966 raft_consensus.cc:2229] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:12.478276 3966 raft_consensus.cc:2258] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:12.478850 3966 ts_tablet_manager.cc:1875] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:12.479380 3965 ts_tablet_manager.cc:1888] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:12.479539 3965 log.cc:1196] T 783db67a272147ed8752c5d6dba4456e P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/783db67a272147ed8752c5d6dba4456e
I20240501 19:57:12.479945 3965 tablet_replica.cc:330] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:12.480053 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 783db67a272147ed8752c5d6dba4456e (table rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]) successfully deleted
I20240501 19:57:12.480094 3965 raft_consensus.cc:2229] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:12.480192 3965 raft_consensus.cc:2258] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:12.480644 3965 ts_tablet_manager.cc:1875] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:12.481097 3966 ts_tablet_manager.cc:1888] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:12.481215 3966 log.cc:1196] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/4c812d2297ad4affa36bdfa865b9498b
I20240501 19:57:12.481489 3966 ts_tablet_manager.cc:1909] T 4c812d2297ad4affa36bdfa865b9498b P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:57:12.482043 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 4c812d2297ad4affa36bdfa865b9498b (table rebalancer_dynamic_table_new_name [id=5943530919264671886a7a554bf2636b]) successfully deleted
I20240501 19:57:12.482266 3965 ts_tablet_manager.cc:1888] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:12.482359 3965 log.cc:1196] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c812d2297ad4affa36bdfa865b9498b
I20240501 19:57:12.482615 3965 ts_tablet_manager.cc:1909] T 4c812d2297ad4affa36bdfa865b9498b P 9d64cdfaeb2d46b29845160e395fc241: Deleting consensus metadata
I20240501 19:57:12.483259 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 4c812d2297ad4affa36bdfa865b9498b (table rebalancer_dynamic_table_new_name [id=5943530919264671886a7a554bf2636b]) successfully deleted
I20240501 19:57:12.558321 3431 consensus_queue.cc:1035] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.562211 4010 consensus_queue.cc:1035] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.576130 24445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:12.582751 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 6a801f93df41406da36c3e37cca08071 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=57f10d93c0c6475c8fb3bbe39557bb43]), partition=
I20240501 19:57:12.582901 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 6a801f93df41406da36c3e37cca08071 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=57f10d93c0c6475c8fb3bbe39557bb43]), partition=
I20240501 19:57:12.582993 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a801f93df41406da36c3e37cca08071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.583123 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a801f93df41406da36c3e37cca08071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.585578 4013 tablet_bootstrap.cc:492] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:12.586606 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 6a801f93df41406da36c3e37cca08071 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=57f10d93c0c6475c8fb3bbe39557bb43]), partition=
I20240501 19:57:12.586858 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a801f93df41406da36c3e37cca08071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.589609 4014 tablet_bootstrap.cc:492] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:12.590588 4014 tablet_bootstrap.cc:654] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.591187 4013 tablet_bootstrap.cc:654] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.592375 4013 tablet_bootstrap.cc:492] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:12.592473 4013 ts_tablet_manager.cc:1367] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:12.592875 4013 raft_consensus.cc:348] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.592973 4013 raft_consensus.cc:374] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.593004 4013 raft_consensus.cc:729] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.593216 4013 consensus_queue.cc:260] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.593557 4013 ts_tablet_manager.cc:1398] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:12.594007 4014 tablet_bootstrap.cc:492] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:12.594141 4014 ts_tablet_manager.cc:1367] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:12.594650 4014 raft_consensus.cc:348] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.594812 4014 raft_consensus.cc:374] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.594894 4014 raft_consensus.cc:729] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.595083 4014 consensus_queue.cc:260] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.595628 4014 ts_tablet_manager.cc:1398] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:12.601116 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:12.612486 27011 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:12.615638 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 63bbbaff2f2e44eea17443f9a3ad3615 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.615639 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 453d72f3b14c471cbd6fcebe50afa8dd (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.615973 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bbbaff2f2e44eea17443f9a3ad3615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.616233 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 63bbbaff2f2e44eea17443f9a3ad3615 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.616480 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bbbaff2f2e44eea17443f9a3ad3615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.618477 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 0f818112217042329c60be020f77df83 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.618721 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f818112217042329c60be020f77df83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.618839 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 453d72f3b14c471cbd6fcebe50afa8dd (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.619096 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453d72f3b14c471cbd6fcebe50afa8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.620836 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 453d72f3b14c471cbd6fcebe50afa8dd (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.621069 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453d72f3b14c471cbd6fcebe50afa8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.621297 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453d72f3b14c471cbd6fcebe50afa8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.623312 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 63bbbaff2f2e44eea17443f9a3ad3615 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.623556 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bbbaff2f2e44eea17443f9a3ad3615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.625553 27406 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:12.627172 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 0f818112217042329c60be020f77df83 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.627413 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f818112217042329c60be020f77df83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.628254 4015 tablet_bootstrap.cc:492] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:12.629779 4031 tablet_bootstrap.cc:492] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:12.634142 4031 tablet_bootstrap.cc:654] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.634611 4015 tablet_bootstrap.cc:654] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.636325 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 0f818112217042329c60be020f77df83 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c43e53fe9e6e47099f9704220e43ab47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:12.636406 4031 tablet_bootstrap.cc:492] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:12.636493 4031 ts_tablet_manager.cc:1367] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 19:57:12.636618 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f818112217042329c60be020f77df83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:12.637002 4031 raft_consensus.cc:348] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.637405 4031 raft_consensus.cc:374] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.637444 4031 raft_consensus.cc:729] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.637564 4031 consensus_queue.cc:260] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.637821 4031 ts_tablet_manager.cc:1398] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:12.638700 4031 tablet_bootstrap.cc:492] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:12.638943 4013 tablet_bootstrap.cc:492] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:12.639539 4031 tablet_bootstrap.cc:654] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.640535 27142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:12.644181 4013 tablet_bootstrap.cc:654] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.653200 24966 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:57:12.665271 24575 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:12.678169 27274 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:12.684024 4030 tablet_bootstrap.cc:492] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:12.685175 4030 tablet_bootstrap.cc:654] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.685887 4032 tablet_bootstrap.cc:492] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:12.686965 4032 tablet_bootstrap.cc:654] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.688627 4031 tablet_bootstrap.cc:492] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:12.688748 4031 ts_tablet_manager.cc:1367] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.001s
I20240501 19:57:12.688954 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:12.689179 4031 raft_consensus.cc:348] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.689311 4031 raft_consensus.cc:374] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.689393 4031 raft_consensus.cc:729] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.689777 4031 consensus_queue.cc:260] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.691823 4031 ts_tablet_manager.cc:1398] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:12.704047 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:12.718514 4013 tablet_bootstrap.cc:492] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:12.718634 4013 ts_tablet_manager.cc:1367] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.000s
I20240501 19:57:12.719259 4013 raft_consensus.cc:348] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.719388 4013 raft_consensus.cc:374] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.719424 4013 raft_consensus.cc:729] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.719539 4013 consensus_queue.cc:260] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.719830 4013 ts_tablet_manager.cc:1398] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:12.724119 4030 tablet_bootstrap.cc:492] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:12.724217 4030 ts_tablet_manager.cc:1367] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20240501 19:57:12.724740 4030 raft_consensus.cc:348] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.724820 4030 raft_consensus.cc:374] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.724839 4030 raft_consensus.cc:729] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.724938 4030 consensus_queue.cc:260] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.725172 4030 ts_tablet_manager.cc:1398] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:12.725291 4030 tablet_bootstrap.cc:492] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:12.726269 4030 tablet_bootstrap.cc:654] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.726475 4032 tablet_bootstrap.cc:492] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:12.726609 4032 ts_tablet_manager.cc:1367] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240501 19:57:12.727160 4032 raft_consensus.cc:348] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:12.727236 4015 tablet_bootstrap.cc:492] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:12.727262 4032 raft_consensus.cc:374] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.727294 4032 raft_consensus.cc:729] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.727325 4015 ts_tablet_manager.cc:1367] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.000s
I20240501 19:57:12.727418 4032 consensus_queue.cc:260] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:12.727702 4030 tablet_bootstrap.cc:492] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:12.727723 4032 ts_tablet_manager.cc:1398] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:12.727778 4030 ts_tablet_manager.cc:1367] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:12.727787 4015 raft_consensus.cc:348] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.727890 4015 raft_consensus.cc:374] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.727921 4015 raft_consensus.cc:729] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.728026 4015 consensus_queue.cc:260] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.728196 4030 raft_consensus.cc:348] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:12.728271 4015 ts_tablet_manager.cc:1398] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:12.728297 4030 raft_consensus.cc:374] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.728335 4030 raft_consensus.cc:729] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.728421 4015 tablet_bootstrap.cc:492] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:12.728463 4030 consensus_queue.cc:260] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:12.728705 4030 ts_tablet_manager.cc:1398] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:12.728850 4030 tablet_bootstrap.cc:492] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:12.729218 4015 tablet_bootstrap.cc:654] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.729517 4030 tablet_bootstrap.cc:654] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.730304 4015 tablet_bootstrap.cc:492] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:12.730377 4015 ts_tablet_manager.cc:1367] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:12.730510 4030 tablet_bootstrap.cc:492] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:12.730571 4030 ts_tablet_manager.cc:1367] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:12.730880 4015 raft_consensus.cc:348] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:12.731083 4015 raft_consensus.cc:374] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.731189 4015 raft_consensus.cc:729] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.731159 4030 raft_consensus.cc:348] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.731364 4030 raft_consensus.cc:374] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.731320 4015 consensus_queue.cc:260] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:12.731427 4030 raft_consensus.cc:729] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.731551 4030 consensus_queue.cc:260] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.731735 4015 ts_tablet_manager.cc:1398] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:12.731824 4030 ts_tablet_manager.cc:1398] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:12.732082 4015 tablet_bootstrap.cc:492] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:12.732795 4015 tablet_bootstrap.cc:654] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:12.736019 4015 tablet_bootstrap.cc:492] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:12.736107 4015 ts_tablet_manager.cc:1367] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:12.736918 27032 raft_consensus.cc:1203] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:12.737712 4015 raft_consensus.cc:348] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.737841 4015 raft_consensus.cc:374] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:12.737881 4015 raft_consensus.cc:729] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:12.738075 4015 consensus_queue.cc:260] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.738346 4015 ts_tablet_manager.cc:1398] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:12.790331 24594 consensus_queue.cc:237] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:12.791364 24857 raft_consensus.cc:1261] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:12.791433 27032 raft_consensus.cc:1261] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:12.792145 2539 consensus_queue.cc:1035] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:12.792295 3264 consensus_queue.cc:1035] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:12.794345 2539 raft_consensus.cc:2932] T 6ca876a9d51e4096955127e7038e2555 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:12.794863 27032 raft_consensus.cc:2932] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:12.795593 24857 raft_consensus.cc:2932] T 6ca876a9d51e4096955127e7038e2555 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:12.798828 24451 catalog_manager.cc:5484] T 6ca876a9d51e4096955127e7038e2555 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:12.802940 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca876a9d51e4096955127e7038e2555 with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:12.803320 4068 tablet_replica.cc:330] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:12.803472 4068 raft_consensus.cc:2229] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:12.803622 4068 raft_consensus.cc:2258] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:12.804056 4068 ts_tablet_manager.cc:1875] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:12.806085 4068 ts_tablet_manager.cc:1888] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:12.806159 4068 log.cc:1196] T 6ca876a9d51e4096955127e7038e2555 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/6ca876a9d51e4096955127e7038e2555
I20240501 19:57:12.806824 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet 6ca876a9d51e4096955127e7038e2555 (table rebalancer_extra_table_16 [id=e4dc82031f3247de9f9b2feb376bfe93]) successfully deleted
I20240501 19:57:12.815006 3623 raft_consensus.cc:482] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:12.815174 3623 raft_consensus.cc:504] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.815546 3623 leader_election.cc:288] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:12.815817 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f818112217042329c60be020f77df83" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:12.815860 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f818112217042329c60be020f77df83" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:12.815953 24725 raft_consensus.cc:2454] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:12.815972 27280 raft_consensus.cc:2454] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:12.816325 24922 leader_election.cc:302] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:12.816493 3623 raft_consensus.cc:2781] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:12.816548 3623 raft_consensus.cc:482] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:12.816591 3623 raft_consensus.cc:3037] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.817487 3623 raft_consensus.cc:504] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.817842 3623 leader_election.cc:288] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:12.818168 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f818112217042329c60be020f77df83" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:12.818293 24725 raft_consensus.cc:3037] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.818770 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f818112217042329c60be020f77df83" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:12.819399 27280 raft_consensus.cc:3037] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.819466 24725 raft_consensus.cc:2454] T 0f818112217042329c60be020f77df83 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:12.819710 24922 leader_election.cc:302] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:12.819845 3286 raft_consensus.cc:2781] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:12.819955 3286 raft_consensus.cc:686] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:12.820084 3286 consensus_queue.cc:237] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.820433 27280 raft_consensus.cc:2454] T 0f818112217042329c60be020f77df83 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:12.822611 24450 catalog_manager.cc:5484] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:12.835134 3264 raft_consensus.cc:482] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:12.835249 3264 raft_consensus.cc:504] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.835741 3264 leader_election.cc:288] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:12.836048 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a801f93df41406da36c3e37cca08071" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:12.836136 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a801f93df41406da36c3e37cca08071" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:12.836200 27423 raft_consensus.cc:2454] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:12.836272 27280 raft_consensus.cc:2454] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:12.836514 24531 leader_election.cc:302] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:12.836861 3264 raft_consensus.cc:2781] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:12.836915 3264 raft_consensus.cc:482] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:12.836956 3264 raft_consensus.cc:3037] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.837813 3264 raft_consensus.cc:504] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.838088 3264 leader_election.cc:288] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:12.838346 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a801f93df41406da36c3e37cca08071" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:12.838418 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a801f93df41406da36c3e37cca08071" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:12.838459 27423 raft_consensus.cc:3037] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.838528 27280 raft_consensus.cc:3037] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.839484 27280 raft_consensus.cc:2454] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:12.839484 27423 raft_consensus.cc:2454] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:12.839752 24531 leader_election.cc:302] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:12.840026 4062 raft_consensus.cc:2781] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:12.840107 4062 raft_consensus.cc:686] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:12.840226 4062 consensus_queue.cc:237] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:12.842588 24450 catalog_manager.cc:5484] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:12.845271 3286 raft_consensus.cc:482] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:12.845369 3286 raft_consensus.cc:504] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.845698 3286 leader_election.cc:288] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:12.845988 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bbbaff2f2e44eea17443f9a3ad3615" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:12.846065 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bbbaff2f2e44eea17443f9a3ad3615" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:12.846129 24595 raft_consensus.cc:2454] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:12.846201 24725 raft_consensus.cc:2454] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:12.846378 24922 leader_election.cc:302] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:12.846661 3286 raft_consensus.cc:2781] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:12.846717 3286 raft_consensus.cc:482] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:12.846753 3286 raft_consensus.cc:3037] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.847790 3286 raft_consensus.cc:504] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.848089 3286 leader_election.cc:288] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:12.848353 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bbbaff2f2e44eea17443f9a3ad3615" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:12.848423 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bbbaff2f2e44eea17443f9a3ad3615" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:12.848474 24595 raft_consensus.cc:3037] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.848532 24725 raft_consensus.cc:3037] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:12.849340 24595 raft_consensus.cc:2454] T 63bbbaff2f2e44eea17443f9a3ad3615 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:12.849612 24922 leader_election.cc:302] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:12.849782 3286 raft_consensus.cc:2781] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:12.849877 3286 raft_consensus.cc:686] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:12.850016 3286 consensus_queue.cc:237] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:12.850199 24725 raft_consensus.cc:2454] T 63bbbaff2f2e44eea17443f9a3ad3615 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:12.852555 24450 catalog_manager.cc:5484] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:12.908972 27423 raft_consensus.cc:1203] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:12.948382 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:12.952198 27280 raft_consensus.cc:1261] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:12.952236 27423 raft_consensus.cc:1261] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:12.952688 4062 consensus_queue.cc:1035] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.952816 3264 consensus_queue.cc:1035] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:12.955682 4116 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:12.955931 4011 mvcc.cc:205] Tried to move back new op lower bound from 7022974701368266752 to 7022974700914855936. Current Snapshot: MvccSnapshot[applied={T|T < 7022974701368266752}]
I20240501 19:57:12.956090 4116 tablet.cc:1620] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:12.956946 4117 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:12.958211 4113 mvcc.cc:205] Tried to move back new op lower bound from 7022974701368266752 to 7022974700914855936. Current Snapshot: MvccSnapshot[applied={T|T < 7022974701368266752}]
I20240501 19:57:12.959177 4118 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:12.958979 24453 catalog_manager.cc:5576] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957) does not have the latest schema for tablet 6a801f93df41406da36c3e37cca08071 (table rebalancer_dynamic_table [id=57f10d93c0c6475c8fb3bbe39557bb43]). Expected version 1 got 0
I20240501 19:57:12.959424 4117 tablet.cc:1620] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:12.959818 4118 tablet.cc:1620] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:12.961622 24454 catalog_manager.cc:5576] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797) does not have the latest schema for tablet 6a801f93df41406da36c3e37cca08071 (table rebalancer_dynamic_table [id=57f10d93c0c6475c8fb3bbe39557bb43]). Expected version 1 got 0
I20240501 19:57:12.962440 24447 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=57f10d93c0c6475c8fb3bbe39557bb43] alter complete (version 1)
I20240501 19:57:13.002962 3424 raft_consensus.cc:482] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:13.003136 3424 raft_consensus.cc:504] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:13.003607 3424 leader_election.cc:288] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:13.003971 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d72f3b14c471cbd6fcebe50afa8dd" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:13.004101 24986 raft_consensus.cc:2454] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:13.004195 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d72f3b14c471cbd6fcebe50afa8dd" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:13.004330 24595 raft_consensus.cc:2454] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:13.004387 27097 leader_election.cc:302] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:13.004549 3424 raft_consensus.cc:2781] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:13.004599 3424 raft_consensus.cc:482] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:13.004639 3424 raft_consensus.cc:3037] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.005620 3424 raft_consensus.cc:504] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:13.005934 3424 leader_election.cc:288] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:13.006244 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d72f3b14c471cbd6fcebe50afa8dd" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:13.006348 24595 raft_consensus.cc:3037] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.006429 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d72f3b14c471cbd6fcebe50afa8dd" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:13.006582 24986 raft_consensus.cc:3037] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.007345 24595 raft_consensus.cc:2454] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:13.007575 24986 raft_consensus.cc:2454] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:13.007611 27096 leader_election.cc:302] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:13.007795 3424 raft_consensus.cc:2781] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:13.007905 3424 raft_consensus.cc:686] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:13.008060 3424 consensus_queue.cc:237] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:13.010578 24453 catalog_manager.cc:5484] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:13.037801 27142 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:13.042001 27011 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:13.043046 27274 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:13.050241 27406 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:13.052201 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:13.054872 24966 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:13.067770 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:13.073668 4116 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:13.073964 4116 tablet.cc:1620] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:13.074839 4118 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:13.075131 4118 tablet.cc:1620] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: MemRowSet was empty: no flush needed.
I20240501 19:57:13.076545 4117 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:13.076821 4117 tablet.cc:1620] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: MemRowSet was empty: no flush needed.
I20240501 19:57:13.080611 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=57f10d93c0c6475c8fb3bbe39557bb43] alter complete (version 2)
I20240501 19:57:13.088184 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:13.088186 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:13.139632 24595 consensus_queue.cc:237] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:13.141031 27424 raft_consensus.cc:1261] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:13.141067 24725 raft_consensus.cc:1261] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:13.141474 3264 consensus_queue.cc:1035] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:13.141582 4062 consensus_queue.cc:1035] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:13.143323 3264 raft_consensus.cc:2932] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:13.143637 24725 raft_consensus.cc:2932] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:13.143673 27424 raft_consensus.cc:2932] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:13.147321 24453 catalog_manager.cc:5484] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:13.150337 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e7bc7629944b3da3fe8fb1ecae32b1 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:13.150905 4150 tablet_replica.cc:330] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:13.151108 4150 raft_consensus.cc:2229] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:13.151280 4150 raft_consensus.cc:2258] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:13.151818 4150 ts_tablet_manager.cc:1875] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:13.154057 4150 ts_tablet_manager.cc:1888] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:13.154120 4150 log.cc:1196] T 39e7bc7629944b3da3fe8fb1ecae32b1 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/39e7bc7629944b3da3fe8fb1ecae32b1
I20240501 19:57:13.154618 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 39e7bc7629944b3da3fe8fb1ecae32b1 (table rebalance_test_table_3 [id=c031f124af6547ff886da7ae7c7f62b0]) successfully deleted
I20240501 19:57:13.187870 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:13.192516 24725 consensus_queue.cc:237] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:13.193686 4116 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:13.194211 27032 raft_consensus.cc:1261] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:13.194752 24857 raft_consensus.cc:1261] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:13.195135 4177 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:13.195314 3440 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:57:13.196080 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb0bb0d3396a436c878f56d157f30eda. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:13.196084 4117 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:13.198601 4118 tablet.cc:1722] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:13.202622 24454 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=57f10d93c0c6475c8fb3bbe39557bb43] alter complete (version 3)
I20240501 19:57:13.203474 3840 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:13.205533 3286 consensus_queue.cc:1035] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:13.205497 24856 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:13.211287 24453 catalog_manager.cc:5484] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:13.212706 27032 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:13.215495 3623 consensus_queue.cc:1035] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:13.309161 24450 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:13.309331 24450 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:13.312788 24450 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 6a801f93df41406da36c3e37cca08071
I20240501 19:57:13.313377 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a801f93df41406da36c3e37cca08071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:13 UTC) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:13.313941 4227 tablet_replica.cc:330] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:13.313977 27406 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a801f93df41406da36c3e37cca08071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:13 UTC) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:13.314451 27274 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a801f93df41406da36c3e37cca08071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:13 UTC) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:13.314771 4229 tablet_replica.cc:330] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:13.314841 4227 raft_consensus.cc:2229] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:13.314980 4229 raft_consensus.cc:2229] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:13.315124 4227 raft_consensus.cc:2258] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:13.315187 4229 raft_consensus.cc:2258] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:13.316466 4229 ts_tablet_manager.cc:1875] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:13.319191 4227 ts_tablet_manager.cc:1875] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:13.320602 4229 ts_tablet_manager.cc:1888] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:13.320729 4229 log.cc:1196] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/6a801f93df41406da36c3e37cca08071
I20240501 19:57:13.321831 4229 ts_tablet_manager.cc:1909] T 6a801f93df41406da36c3e37cca08071 P 53e34bb09d8e41129a5683079ff63610: Deleting consensus metadata
I20240501 19:57:13.322754 4227 ts_tablet_manager.cc:1888] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:13.322928 4227 log.cc:1196] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a801f93df41406da36c3e37cca08071
I20240501 19:57:13.323324 4227 ts_tablet_manager.cc:1909] T 6a801f93df41406da36c3e37cca08071 P bbf7735f82ca4f65a5170ac7909faa47: Deleting consensus metadata
I20240501 19:57:13.324025 4228 tablet_replica.cc:330] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:13.324211 4228 raft_consensus.cc:2229] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:13.324304 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 6a801f93df41406da36c3e37cca08071 (table rebalancer_dynamic_table_new_name [id=57f10d93c0c6475c8fb3bbe39557bb43]) successfully deleted
I20240501 19:57:13.324434 4228 raft_consensus.cc:2258] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:13.324987 4228 ts_tablet_manager.cc:1875] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:13.327955 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 6a801f93df41406da36c3e37cca08071 (table rebalancer_dynamic_table_new_name [id=57f10d93c0c6475c8fb3bbe39557bb43]) successfully deleted
I20240501 19:57:13.338869 4228 ts_tablet_manager.cc:1888] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:13.339017 4228 log.cc:1196] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/6a801f93df41406da36c3e37cca08071
I20240501 19:57:13.339393 4228 ts_tablet_manager.cc:1909] T 6a801f93df41406da36c3e37cca08071 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting consensus metadata
I20240501 19:57:13.340180 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 6a801f93df41406da36c3e37cca08071 (table rebalancer_dynamic_table_new_name [id=57f10d93c0c6475c8fb3bbe39557bb43]) successfully deleted
I20240501 19:57:13.356848 3623 consensus_queue.cc:1035] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:13.362452 27011 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:13.362952 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:13.363147 3286 consensus_queue.cc:1035] T 63bbbaff2f2e44eea17443f9a3ad3615 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:13.369338 27142 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:13.374146 24966 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:13.393600 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:13.403204 27274 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:13.405858 24575 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:13.406559 27406 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:13.418545 3424 consensus_queue.cc:1035] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:13.419085 24450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:13.425817 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 2e98a24a7cee475ab6ba8985a677035f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67262ce2cbcc4871b3d4d673699b87a1]), partition=
I20240501 19:57:13.426043 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 2e98a24a7cee475ab6ba8985a677035f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67262ce2cbcc4871b3d4d673699b87a1]), partition=
I20240501 19:57:13.426100 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e98a24a7cee475ab6ba8985a677035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.426247 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e98a24a7cee475ab6ba8985a677035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.427531 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 2e98a24a7cee475ab6ba8985a677035f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67262ce2cbcc4871b3d4d673699b87a1]), partition=
I20240501 19:57:13.427767 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e98a24a7cee475ab6ba8985a677035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.429844 4247 tablet_bootstrap.cc:492] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:13.431356 4247 tablet_bootstrap.cc:654] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.433377 4247 tablet_bootstrap.cc:492] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:13.433574 4247 ts_tablet_manager.cc:1367] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:13.434477 4247 raft_consensus.cc:348] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.434741 4247 raft_consensus.cc:374] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.434820 4247 raft_consensus.cc:729] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.435099 4247 consensus_queue.cc:260] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.436789 4247 ts_tablet_manager.cc:1398] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:13.440804 4246 tablet_bootstrap.cc:492] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:13.441237 4040 consensus_queue.cc:1035] T 453d72f3b14c471cbd6fcebe50afa8dd P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:13.441690 4246 tablet_bootstrap.cc:654] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.443301 4245 tablet_bootstrap.cc:492] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:13.443573 4010 raft_consensus.cc:482] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:13.443660 4010 raft_consensus.cc:504] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.444008 4010 leader_election.cc:288] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:13.444406 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e98a24a7cee475ab6ba8985a677035f" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:13.444535 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e98a24a7cee475ab6ba8985a677035f" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
W20240501 19:57:13.444759 26965 leader_election.cc:341] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:57:13.445047 26967 leader_election.cc:341] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:13.445125 26967 leader_election.cc:302] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971
I20240501 19:57:13.445469 4010 raft_consensus.cc:2726] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:13.446012 4245 tablet_bootstrap.cc:654] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.446915 4246 tablet_bootstrap.cc:492] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:13.447098 4246 ts_tablet_manager.cc:1367] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:13.447942 4246 raft_consensus.cc:348] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.448088 4246 raft_consensus.cc:374] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.448153 4246 raft_consensus.cc:729] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.448325 4246 consensus_queue.cc:260] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.448766 4245 tablet_bootstrap.cc:492] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:13.448848 4245 ts_tablet_manager.cc:1367] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:13.449481 4246 ts_tablet_manager.cc:1398] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:13.449925 4245 raft_consensus.cc:348] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.450007 4245 raft_consensus.cc:374] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.450042 4245 raft_consensus.cc:729] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.450156 4245 consensus_queue.cc:260] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.450381 4245 ts_tablet_manager.cc:1398] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:13.577327 4040 raft_consensus.cc:482] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:13.577458 4040 raft_consensus.cc:504] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.577847 4040 leader_election.cc:288] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:13.578263 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e98a24a7cee475ab6ba8985a677035f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:13.578310 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e98a24a7cee475ab6ba8985a677035f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:13.578410 27031 raft_consensus.cc:2454] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:13.578454 24725 raft_consensus.cc:2454] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:13.579149 27096 leader_election.cc:302] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:57:13.579454 4040 raft_consensus.cc:2781] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:13.579509 4040 raft_consensus.cc:482] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:13.579543 4040 raft_consensus.cc:3037] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.581004 4040 raft_consensus.cc:504] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.581317 4040 leader_election.cc:288] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:13.582559 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e98a24a7cee475ab6ba8985a677035f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:13.582675 24725 raft_consensus.cc:3037] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.583681 24725 raft_consensus.cc:2454] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:13.584935 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e98a24a7cee475ab6ba8985a677035f" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:13.585031 27031 raft_consensus.cc:3037] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.585973 27031 raft_consensus.cc:2454] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:13.586350 27096 leader_election.cc:302] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1; no voters:
I20240501 19:57:13.588750 4040 raft_consensus.cc:2781] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:13.588833 4040 raft_consensus.cc:686] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:13.588950 4040 consensus_queue.cc:237] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:13.595322 24451 catalog_manager.cc:5484] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:13.598156 24445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:13.605621 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 1647b75a7f8f4a28ae66b85d3fbfeb4c (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.605913 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1647b75a7f8f4a28ae66b85d3fbfeb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.607010 27274 tablet_service.cc:1467] Processing CreateTablet for tablet 40bff70a9c3447c7ab8045568c779c78 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.607223 27267 tablet_service.cc:1467] Processing CreateTablet for tablet ea7bbf2e0e5f48ed959195deb548b051 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.607558 27406 tablet_service.cc:1467] Processing CreateTablet for tablet 1647b75a7f8f4a28ae66b85d3fbfeb4c (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.607823 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1647b75a7f8f4a28ae66b85d3fbfeb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.608644 4246 tablet_bootstrap.cc:492] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:13.608839 27404 tablet_service.cc:1467] Processing CreateTablet for tablet 40bff70a9c3447c7ab8045568c779c78 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.608964 27405 tablet_service.cc:1467] Processing CreateTablet for tablet ea7bbf2e0e5f48ed959195deb548b051 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.609076 27404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bff70a9c3447c7ab8045568c779c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.609208 27405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7bbf2e0e5f48ed959195deb548b051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.609647 4246 tablet_bootstrap.cc:654] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.610332 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 40bff70a9c3447c7ab8045568c779c78 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.610400 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 1647b75a7f8f4a28ae66b85d3fbfeb4c (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.610563 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bff70a9c3447c7ab8045568c779c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.610649 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1647b75a7f8f4a28ae66b85d3fbfeb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.611249 27012 tablet_service.cc:1467] Processing CreateTablet for tablet ea7bbf2e0e5f48ed959195deb548b051 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=ccb61fad3b764bc5af0b6764b6bb79cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:13.611506 27012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7bbf2e0e5f48ed959195deb548b051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.613647 4247 tablet_bootstrap.cc:492] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:13.607280 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bff70a9c3447c7ab8045568c779c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.614552 4247 tablet_bootstrap.cc:654] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.615563 4247 tablet_bootstrap.cc:492] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:13.615584 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7bbf2e0e5f48ed959195deb548b051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.615638 4247 ts_tablet_manager.cc:1367] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:13.616087 4247 raft_consensus.cc:348] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.616197 4247 raft_consensus.cc:374] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.616225 4247 raft_consensus.cc:729] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.616318 4247 consensus_queue.cc:260] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.616591 4247 ts_tablet_manager.cc:1398] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:13.616741 4247 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:13.617487 4247 tablet_bootstrap.cc:654] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.622045 4308 tablet_bootstrap.cc:492] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:13.623071 4308 tablet_bootstrap.cc:654] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.626451 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:13.628310 4306 tablet_bootstrap.cc:492] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:13.628903 4310 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:13.630263 4246 tablet_bootstrap.cc:492] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:13.630347 4246 ts_tablet_manager.cc:1367] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 19:57:13.630821 4246 raft_consensus.cc:348] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.630939 4246 raft_consensus.cc:374] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.630985 4246 raft_consensus.cc:729] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.631111 4246 consensus_queue.cc:260] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.631418 4246 ts_tablet_manager.cc:1398] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:13.632957 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:13.636534 4308 tablet_bootstrap.cc:492] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:13.636659 4308 ts_tablet_manager.cc:1367] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:57:13.637127 4308 raft_consensus.cc:348] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.637224 4308 raft_consensus.cc:374] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.637254 4308 raft_consensus.cc:729] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.637362 4308 consensus_queue.cc:260] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.637619 4308 ts_tablet_manager.cc:1398] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:13.637854 27012 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:13.637988 4306 tablet_bootstrap.cc:654] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.642344 24966 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:13.645289 3264 raft_consensus.cc:482] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:13.645362 3264 raft_consensus.cc:504] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.645701 3264 leader_election.cc:288] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:13.645956 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1647b75a7f8f4a28ae66b85d3fbfeb4c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
W20240501 19:57:13.646270 24531 leader_election.cc:341] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:13.647133 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1647b75a7f8f4a28ae66b85d3fbfeb4c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:13.647238 24725 raft_consensus.cc:2454] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:13.647244 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:13.647725 24529 leader_election.cc:302] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:13.647909 2539 raft_consensus.cc:2781] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:13.647948 2539 raft_consensus.cc:482] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:13.647979 2539 raft_consensus.cc:3037] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.648944 2539 raft_consensus.cc:504] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.649264 2539 leader_election.cc:288] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:13.649837 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1647b75a7f8f4a28ae66b85d3fbfeb4c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:13.649943 24725 raft_consensus.cc:3037] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.650866 24725 raft_consensus.cc:2454] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:13.653398 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1647b75a7f8f4a28ae66b85d3fbfeb4c" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
W20240501 19:57:13.653729 24531 leader_election.cc:341] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:13.654161 24529 leader_election.cc:302] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:13.655860 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:13.655920 2539 raft_consensus.cc:2781] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:13.655992 2539 raft_consensus.cc:686] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:13.656105 2539 consensus_queue.cc:237] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.658308 24453 catalog_manager.cc:5484] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:13.659852 4310 tablet_bootstrap.cc:654] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.660784 4247 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1: No bootstrap required, opened a new log
I20240501 19:57:13.660879 4247 ts_tablet_manager.cc:1367] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.000s
I20240501 19:57:13.661309 4247 raft_consensus.cc:348] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.661573 4247 raft_consensus.cc:374] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.661665 4247 raft_consensus.cc:729] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.661902 4247 consensus_queue.cc:260] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.662418 4247 ts_tablet_manager.cc:1398] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:13.671187 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:13.683387 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:13.685266 4306 tablet_bootstrap.cc:492] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:13.685408 4306 ts_tablet_manager.cc:1367] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.000s
I20240501 19:57:13.691401 4306 raft_consensus.cc:348] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.698969 4306 raft_consensus.cc:374] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.699056 4306 raft_consensus.cc:729] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.699188 4306 consensus_queue.cc:260] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.699571 4306 ts_tablet_manager.cc:1398] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.014s user 0.000s sys 0.001s
I20240501 19:57:13.700582 4306 tablet_bootstrap.cc:492] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:13.702940 4306 tablet_bootstrap.cc:654] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.709707 4316 ts_tablet_manager.cc:902] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:13.717967 4316 tablet_copy_client.cc:308] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:57:13.718571 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet cb0bb0d3396a436c878f56d157f30eda from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:35875)
I20240501 19:57:13.718629 4306 tablet_bootstrap.cc:492] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:13.718683 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet cb0bb0d3396a436c878f56d157f30eda from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:35875: session id = 29d1a658e2b946e0b412e5fef82db9d9-cb0bb0d3396a436c878f56d157f30eda
I20240501 19:57:13.718736 4306 ts_tablet_manager.cc:1367] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240501 19:57:13.719198 4306 raft_consensus.cc:348] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.719323 4306 raft_consensus.cc:374] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.719353 4306 raft_consensus.cc:729] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.719470 4306 consensus_queue.cc:260] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.719710 24746 tablet_copy_source_session.cc:205] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:13.719766 4306 ts_tablet_manager.cc:1398] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:13.719930 4306 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:13.720633 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0bb0d3396a436c878f56d157f30eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:13.720837 4306 tablet_bootstrap.cc:654] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.721827 4310 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:13.721915 4310 ts_tablet_manager.cc:1367] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.000s
I20240501 19:57:13.722491 4310 raft_consensus.cc:348] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.722653 4310 raft_consensus.cc:374] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.722692 4310 raft_consensus.cc:729] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.722828 4310 consensus_queue.cc:260] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.723062 4306 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561: No bootstrap required, opened a new log
I20240501 19:57:13.723147 4306 ts_tablet_manager.cc:1367] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:13.723148 4310 ts_tablet_manager.cc:1398] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:13.723310 4310 tablet_bootstrap.cc:492] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:13.723548 4306 raft_consensus.cc:348] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.723637 4306 raft_consensus.cc:374] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.723664 4306 raft_consensus.cc:729] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.723773 4306 consensus_queue.cc:260] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.724004 4306 ts_tablet_manager.cc:1398] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:13.724188 4316 tablet_copy_client.cc:783] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:13.724277 4310 tablet_bootstrap.cc:654] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:13.724337 4316 tablet_copy_client.cc:647] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:13.726292 3848 raft_consensus.cc:482] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:13.726369 3848 raft_consensus.cc:504] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.726707 3848 leader_election.cc:288] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:13.727177 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7bbf2e0e5f48ed959195deb548b051" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
I20240501 19:57:13.727308 27031 raft_consensus.cc:2454] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:13.727551 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7bbf2e0e5f48ed959195deb548b051" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:13.727672 27423 raft_consensus.cc:2454] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 0.
I20240501 19:57:13.727838 27227 leader_election.cc:302] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:13.728302 4310 tablet_bootstrap.cc:492] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:13.728339 3848 raft_consensus.cc:2781] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:13.728407 3848 raft_consensus.cc:482] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:13.728456 3848 raft_consensus.cc:3037] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.728526 4310 ts_tablet_manager.cc:1367] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:13.729215 4310 raft_consensus.cc:348] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.729355 4310 raft_consensus.cc:374] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:13.729422 4310 raft_consensus.cc:729] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.729566 4310 consensus_queue.cc:260] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.729626 3848 raft_consensus.cc:504] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.729900 4310 ts_tablet_manager.cc:1398] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:13.729990 3848 leader_election.cc:288] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:13.730727 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7bbf2e0e5f48ed959195deb548b051" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:13.730846 27423 raft_consensus.cc:3037] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.730918 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7bbf2e0e5f48ed959195deb548b051" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:13.731025 27031 raft_consensus.cc:3037] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.731871 27423 raft_consensus.cc:2454] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:13.732029 27031 raft_consensus.cc:2454] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 1.
I20240501 19:57:13.732203 27229 leader_election.cc:302] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:13.732376 4018 raft_consensus.cc:2781] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:13.732472 4018 raft_consensus.cc:686] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:13.732600 4018 consensus_queue.cc:237] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.735700 4316 tablet_copy_client.cc:523] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:13.735996 24445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:13.737071 4316 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:13.738972 24453 catalog_manager.cc:5484] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:13.746161 24725 raft_consensus.cc:1261] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:13.746408 27031 raft_consensus.cc:1261] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:13.746783 3424 consensus_queue.cc:1035] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:13.747076 3424 consensus_queue.cc:1035] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:13.750286 24445 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 2e98a24a7cee475ab6ba8985a677035f (table rebalancer_dynamic_table [id=67262ce2cbcc4871b3d4d673699b87a1]). Expected version 1 got 0
I20240501 19:57:13.751900 24445 catalog_manager.cc:5576] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247) does not have the latest schema for tablet 2e98a24a7cee475ab6ba8985a677035f (table rebalancer_dynamic_table [id=67262ce2cbcc4871b3d4d673699b87a1]). Expected version 1 got 0
I20240501 19:57:13.753934 4335 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:13.755280 4335 tablet.cc:1620] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:13.756758 24453 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet 2e98a24a7cee475ab6ba8985a677035f (table rebalancer_dynamic_table [id=67262ce2cbcc4871b3d4d673699b87a1]). Expected version 1 got 0
I20240501 19:57:13.758204 4337 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:13.758498 4337 tablet.cc:1620] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:13.763935 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=67262ce2cbcc4871b3d4d673699b87a1] alter complete (version 1)
I20240501 19:57:13.765854 4341 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:13.766292 4341 tablet.cc:1620] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:13.786129 4316 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:13.786432 4316 tablet_bootstrap.cc:492] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:57:13.786711 4316 ts_tablet_manager.cc:1367] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.050s user 0.027s sys 0.000s
I20240501 19:57:13.787149 4316 raft_consensus.cc:348] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:13.787269 4316 raft_consensus.cc:729] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:13.787410 4316 consensus_queue.cc:260] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:13.787819 4316 ts_tablet_manager.cc:1398] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:13.789121 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-cb0bb0d3396a436c878f56d157f30eda received from {username='slave'} at 127.17.65.6:35875
I20240501 19:57:13.789211 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-cb0bb0d3396a436c878f56d157f30eda on tablet cb0bb0d3396a436c878f56d157f30eda with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:13.797149 24595 consensus_queue.cc:237] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:13.798334 27031 raft_consensus.cc:1261] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:13.798662 24987 raft_consensus.cc:1261] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:13.798699 2539 consensus_queue.cc:1035] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:13.799513 2539 consensus_queue.cc:1035] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:13.801731 3264 raft_consensus.cc:2932] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:13.802632 24986 raft_consensus.cc:2932] T c3d31791486b4e68a9e5283faadb3f8c P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:13.803745 27031 raft_consensus.cc:2932] T c3d31791486b4e68a9e5283faadb3f8c P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:13.806231 24451 catalog_manager.cc:5484] T c3d31791486b4e68a9e5283faadb3f8c P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 4 to 5, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:13.809170 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d31791486b4e68a9e5283faadb3f8c with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:13.809513 4352 tablet_replica.cc:330] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:13.809667 4352 raft_consensus.cc:2229] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:13.809815 4352 raft_consensus.cc:2258] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:13.810297 4352 ts_tablet_manager.cc:1875] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:13.813349 4352 ts_tablet_manager.cc:1888] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:13.813438 4352 log.cc:1196] T c3d31791486b4e68a9e5283faadb3f8c P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3d31791486b4e68a9e5283faadb3f8c
I20240501 19:57:13.814894 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet c3d31791486b4e68a9e5283faadb3f8c (table rebalance_test_table_9 [id=5c03ef128ef544e1bdcb56a4e3d4c2de]) successfully deleted
I20240501 19:57:13.869652 24446 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:57:13.875959 4243 raft_consensus.cc:482] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:13.876076 4243 raft_consensus.cc:504] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:13.876492 4243 leader_election.cc:288] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:13.877177 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1647b75a7f8f4a28ae66b85d3fbfeb4c" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:13.877489 4335 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:13.877795 4335 tablet.cc:1620] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: MemRowSet was empty: no flush needed.
I20240501 19:57:13.879261 4337 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:13.879551 4337 tablet.cc:1620] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:13.882406 4341 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:57:13.882696 4341 tablet.cc:1620] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: MemRowSet was empty: no flush needed.
I20240501 19:57:13.882907 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1647b75a7f8f4a28ae66b85d3fbfeb4c" candidate_uuid: "4a6e54e86a9949e5970b42e8e32e7561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:13.883095 24727 raft_consensus.cc:2379] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a6e54e86a9949e5970b42e8e32e7561 in current term 1: Already voted for candidate bbf7735f82ca4f65a5170ac7909faa47 in this term.
I20240501 19:57:13.883531 27359 leader_election.cc:302] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6e54e86a9949e5970b42e8e32e7561; no voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:13.883930 4243 raft_consensus.cc:3037] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:13.885015 4243 raft_consensus.cc:2726] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:13.886051 24451 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=67262ce2cbcc4871b3d4d673699b87a1] alter complete (version 2)
I20240501 19:57:13.987751 24453 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:57:13.995725 4335 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:13.998225 4337 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:14.001948 4341 tablet.cc:1722] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:57:14.008018 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table_new_name [id=67262ce2cbcc4871b3d4d673699b87a1] alter complete (version 3)
I20240501 19:57:14.040766 3431 raft_consensus.cc:482] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:14.040884 3431 raft_consensus.cc:504] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:14.041206 3431 leader_election.cc:288] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:14.041501 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bff70a9c3447c7ab8045568c779c78" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:14.041538 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bff70a9c3447c7ab8045568c779c78" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561" is_pre_election: true
I20240501 19:57:14.041616 27278 raft_consensus.cc:2454] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:14.041698 27423 raft_consensus.cc:2454] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 0.
I20240501 19:57:14.041914 26967 leader_election.cc:302] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:14.042078 3431 raft_consensus.cc:2781] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:14.042140 3431 raft_consensus.cc:482] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:14.042184 3431 raft_consensus.cc:3037] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.043253 3431 raft_consensus.cc:504] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:14.043536 3431 leader_election.cc:288] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:14.043816 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bff70a9c3447c7ab8045568c779c78" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:14.043845 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bff70a9c3447c7ab8045568c779c78" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:14.043903 27278 raft_consensus.cc:3037] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.043944 27423 raft_consensus.cc:3037] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.044823 27278 raft_consensus.cc:2454] T 40bff70a9c3447c7ab8045568c779c78 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:14.044942 27423 raft_consensus.cc:2454] T 40bff70a9c3447c7ab8045568c779c78 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:14.045167 26967 leader_election.cc:302] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561; no voters:
I20240501 19:57:14.045308 3431 raft_consensus.cc:2781] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:14.045395 3431 raft_consensus.cc:686] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:14.045532 3431 consensus_queue.cc:237] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:14.047631 24453 catalog_manager.cc:5484] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:14.064903 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:14.069703 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:14.071182 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:14.072666 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:14.082477 24966 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:14.086337 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:14.101926 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:14.112366 24451 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:14.112530 24451 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:57:14.114713 24451 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: Sending DeleteTablet for 3 replicas of tablet 2e98a24a7cee475ab6ba8985a677035f
I20240501 19:57:14.115284 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e98a24a7cee475ab6ba8985a677035f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:14 UTC) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:14.115602 4421 tablet_replica.cc:330] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:14.115828 4421 raft_consensus.cc:2229] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:14.115821 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e98a24a7cee475ab6ba8985a677035f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:14 UTC) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:14.116029 4421 raft_consensus.cc:2258] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:14.116631 4421 ts_tablet_manager.cc:1875] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:14.116710 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:14.118592 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e98a24a7cee475ab6ba8985a677035f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:14 UTC) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:14.118929 4423 tablet_replica.cc:330] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:14.119019 4421 ts_tablet_manager.cc:1888] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:14.119146 4421 log.cc:1196] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/2e98a24a7cee475ab6ba8985a677035f
I20240501 19:57:14.119433 4421 ts_tablet_manager.cc:1909] T 2e98a24a7cee475ab6ba8985a677035f P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting consensus metadata
I20240501 19:57:14.120597 4423 raft_consensus.cc:2229] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:14.120775 4423 raft_consensus.cc:2258] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:14.121258 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 2e98a24a7cee475ab6ba8985a677035f (table rebalancer_dynamic_table_new_name [id=67262ce2cbcc4871b3d4d673699b87a1]) successfully deleted
I20240501 19:57:14.121574 4423 ts_tablet_manager.cc:1875] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:14.124133 4423 ts_tablet_manager.cc:1888] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:14.124197 4423 log.cc:1196] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e98a24a7cee475ab6ba8985a677035f
I20240501 19:57:14.124517 4423 ts_tablet_manager.cc:1909] T 2e98a24a7cee475ab6ba8985a677035f P 9081292597be4a72a42b216fddcd7971: Deleting consensus metadata
I20240501 19:57:14.125329 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 2e98a24a7cee475ab6ba8985a677035f (table rebalancer_dynamic_table_new_name [id=67262ce2cbcc4871b3d4d673699b87a1]) successfully deleted
I20240501 19:57:14.135191 4422 tablet_replica.cc:330] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:14.135336 4422 raft_consensus.cc:2229] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:14.135517 4422 raft_consensus.cc:2258] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:14.136092 4422 ts_tablet_manager.cc:1875] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:14.137912 4422 ts_tablet_manager.cc:1888] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:57:14.137982 4422 log.cc:1196] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/2e98a24a7cee475ab6ba8985a677035f
I20240501 19:57:14.138640 4422 ts_tablet_manager.cc:1909] T 2e98a24a7cee475ab6ba8985a677035f P 29d1a658e2b946e0b412e5fef82db9d9: Deleting consensus metadata
I20240501 19:57:14.139318 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 2e98a24a7cee475ab6ba8985a677035f (table rebalancer_dynamic_table_new_name [id=67262ce2cbcc4871b3d4d673699b87a1]) successfully deleted
I20240501 19:57:14.149952 4350 consensus_queue.cc:1035] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:14.156684 2539 consensus_queue.cc:1035] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:14.196274 4018 consensus_queue.cc:1035] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:14.203714 4018 consensus_queue.cc:1035] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:14.217244 24451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:57:14.224447 24836 tablet_service.cc:1467] Processing CreateTablet for tablet c7a53f32ce714398ac2bb7e9878e4383 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=176edbc9ab894af8bd65c3416fd97e32]), partition=
I20240501 19:57:14.224712 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a53f32ce714398ac2bb7e9878e4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.227571 24575 tablet_service.cc:1467] Processing CreateTablet for tablet c7a53f32ce714398ac2bb7e9878e4383 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=176edbc9ab894af8bd65c3416fd97e32]), partition=
I20240501 19:57:14.227839 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a53f32ce714398ac2bb7e9878e4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.227839 24707 tablet_service.cc:1467] Processing CreateTablet for tablet c7a53f32ce714398ac2bb7e9878e4383 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=176edbc9ab894af8bd65c3416fd97e32]), partition=
I20240501 19:57:14.228060 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a53f32ce714398ac2bb7e9878e4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.230595 4478 tablet_bootstrap.cc:492] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:14.231834 4478 tablet_bootstrap.cc:654] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.233114 4478 tablet_bootstrap.cc:492] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:14.233215 4478 ts_tablet_manager.cc:1367] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:14.233726 4478 raft_consensus.cc:348] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.233875 4478 raft_consensus.cc:374] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.233927 4478 raft_consensus.cc:729] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.234050 4478 consensus_queue.cc:260] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.234405 4478 ts_tablet_manager.cc:1398] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.235669 4481 tablet_bootstrap.cc:492] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:14.236572 4481 tablet_bootstrap.cc:654] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.239472 4479 tablet_bootstrap.cc:492] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:14.241065 4481 tablet_bootstrap.cc:492] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:14.241165 4481 ts_tablet_manager.cc:1367] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:14.241961 4479 tablet_bootstrap.cc:654] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.249095 4479 tablet_bootstrap.cc:492] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47: No bootstrap required, opened a new log
I20240501 19:57:14.249228 4479 ts_tablet_manager.cc:1367] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:57:14.250478 4479 raft_consensus.cc:348] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.250579 4479 raft_consensus.cc:374] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.250614 4479 raft_consensus.cc:729] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.250749 4479 consensus_queue.cc:260] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.251158 4479 ts_tablet_manager.cc:1398] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:14.263386 4481 raft_consensus.cc:348] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.264310 4481 raft_consensus.cc:374] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.264381 4481 raft_consensus.cc:729] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.265162 4481 consensus_queue.cc:260] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.267573 4481 ts_tablet_manager.cc:1398] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20240501 19:57:14.299544 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:14.304374 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:14.304684 27162 raft_consensus.cc:1203] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:57:14.309988 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:14.319769 24966 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:14.323618 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:14.334832 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:14.339238 27142 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:14.342895 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:14.405768 24727 consensus_queue.cc:237] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:14.411423 27163 raft_consensus.cc:1261] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:14.411592 27032 raft_consensus.cc:1261] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:14.412065 4180 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:14.412345 4180 consensus_queue.cc:1035] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:14.414027 4180 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:14.414501 27032 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:14.417407 24447 catalog_manager.cc:5484] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 6 to 7, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:14.420221 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0bb0d3396a436c878f56d157f30eda with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:14.420622 4518 tablet_replica.cc:330] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:14.420822 4518 raft_consensus.cc:2229] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:14.421523 4518 raft_consensus.cc:2258] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:14.422168 4518 ts_tablet_manager.cc:1875] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:14.424795 4518 ts_tablet_manager.cc:1888] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:14.424875 4518 log.cc:1196] T cb0bb0d3396a436c878f56d157f30eda P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/cb0bb0d3396a436c878f56d157f30eda
I20240501 19:57:14.424772 27162 raft_consensus.cc:2932] T cb0bb0d3396a436c878f56d157f30eda P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:14.425384 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet cb0bb0d3396a436c878f56d157f30eda (table rebalance_test_table_5 [id=d3b635ed6a7f49d19cd24a7efc6f99d8]) successfully deleted
I20240501 19:57:14.541129 4010 consensus_queue.cc:1035] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:14.544777 3431 consensus_queue.cc:1035] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:14.591280 4350 raft_consensus.cc:482] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:14.591426 4350 raft_consensus.cc:504] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.591835 4350 leader_election.cc:288] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:14.592345 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a53f32ce714398ac2bb7e9878e4383" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:14.592471 24727 raft_consensus.cc:2454] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:14.592695 24529 leader_election.cc:302] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:14.592764 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a53f32ce714398ac2bb7e9878e4383" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:14.592852 4350 raft_consensus.cc:2781] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:14.592887 24856 raft_consensus.cc:2454] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 0.
I20240501 19:57:14.592905 4350 raft_consensus.cc:482] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:14.592952 4350 raft_consensus.cc:3037] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.593873 4350 raft_consensus.cc:504] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.594189 4350 leader_election.cc:288] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:14.594445 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a53f32ce714398ac2bb7e9878e4383" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:14.594571 24727 raft_consensus.cc:3037] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.594620 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a53f32ce714398ac2bb7e9878e4383" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:14.594744 24856 raft_consensus.cc:3037] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.595670 24727 raft_consensus.cc:2454] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:14.595927 24529 leader_election.cc:302] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:14.596094 4350 raft_consensus.cc:2781] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:14.596174 4350 raft_consensus.cc:686] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:14.596397 4350 consensus_queue.cc:237] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:14.596603 24856 raft_consensus.cc:2454] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:14.599684 24447 catalog_manager.cc:5484] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:14.628129 24447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36710:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:14.634615 24707 tablet_service.cc:1467] Processing CreateTablet for tablet be20e5addc5f43c9855df6913d67d17b (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.634641 24836 tablet_service.cc:1467] Processing CreateTablet for tablet be20e5addc5f43c9855df6913d67d17b (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.634953 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be20e5addc5f43c9855df6913d67d17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.634956 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be20e5addc5f43c9855df6913d67d17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.634932 27267 tablet_service.cc:1467] Processing CreateTablet for tablet c5b0c2b7abe74eb8b8bf30d8b3cf756d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.635196 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b0c2b7abe74eb8b8bf30d8b3cf756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.635345 24966 tablet_service.cc:1467] Processing CreateTablet for tablet be20e5addc5f43c9855df6913d67d17b (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.635607 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be20e5addc5f43c9855df6913d67d17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.637311 4478 tablet_bootstrap.cc:492] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:14.637526 4481 tablet_bootstrap.cc:492] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:14.638255 4478 tablet_bootstrap.cc:654] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.638438 4481 tablet_bootstrap.cc:654] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.638438 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 99ffec91a9d34152b7bc71051ae59c43 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.638695 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ffec91a9d34152b7bc71051ae59c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.638974 27142 tablet_service.cc:1467] Processing CreateTablet for tablet c5b0c2b7abe74eb8b8bf30d8b3cf756d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.639225 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b0c2b7abe74eb8b8bf30d8b3cf756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.640966 4538 tablet_bootstrap.cc:492] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:14.641798 4538 tablet_bootstrap.cc:654] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.642232 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 99ffec91a9d34152b7bc71051ae59c43 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.642472 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ffec91a9d34152b7bc71051ae59c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.642959 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 99ffec91a9d34152b7bc71051ae59c43 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.643219 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ffec91a9d34152b7bc71051ae59c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.644671 4534 tablet_bootstrap.cc:492] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:14.645503 4534 tablet_bootstrap.cc:654] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.646559 4481 tablet_bootstrap.cc:492] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241: No bootstrap required, opened a new log
I20240501 19:57:14.646624 4534 tablet_bootstrap.cc:492] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610: No bootstrap required, opened a new log
I20240501 19:57:14.646653 4481 ts_tablet_manager.cc:1367] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:57:14.646755 4534 ts_tablet_manager.cc:1367] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:14.647064 4481 raft_consensus.cc:348] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.647168 4481 raft_consensus.cc:374] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.647210 4481 raft_consensus.cc:729] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.647330 4481 consensus_queue.cc:260] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.647233 4534 raft_consensus.cc:348] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:14.647485 4534 raft_consensus.cc:374] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.647526 4534 raft_consensus.cc:729] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.647647 4481 ts_tablet_manager.cc:1398] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.647639 4534 consensus_queue.cc:260] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:14.648005 4534 ts_tablet_manager.cc:1398] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.650432 4478 tablet_bootstrap.cc:492] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:14.650525 4478 ts_tablet_manager.cc:1367] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:57:14.650997 4478 raft_consensus.cc:348] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.651131 4478 raft_consensus.cc:374] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.651180 4478 raft_consensus.cc:729] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.651293 4478 consensus_queue.cc:260] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.651571 4478 ts_tablet_manager.cc:1398] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.651742 4478 tablet_bootstrap.cc:492] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:14.651908 24707 tablet_service.cc:1467] Processing CreateTablet for tablet c5b0c2b7abe74eb8b8bf30d8b3cf756d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c5471c2378ba42119681aafa1ad837ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:14.652104 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b0c2b7abe74eb8b8bf30d8b3cf756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.652616 4478 tablet_bootstrap.cc:654] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.653573 4478 tablet_bootstrap.cc:492] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:14.653645 4478 ts_tablet_manager.cc:1367] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:14.654266 4478 raft_consensus.cc:348] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.654404 4478 raft_consensus.cc:374] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.654464 4478 raft_consensus.cc:729] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.654594 4478 consensus_queue.cc:260] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.654906 4478 ts_tablet_manager.cc:1398] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.655140 4478 tablet_bootstrap.cc:492] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:14.656250 4478 tablet_bootstrap.cc:654] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.657714 4478 tablet_bootstrap.cc:492] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971: No bootstrap required, opened a new log
I20240501 19:57:14.657843 4478 ts_tablet_manager.cc:1367] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:14.658277 4478 raft_consensus.cc:348] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:14.658528 4478 raft_consensus.cc:374] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.658574 4478 raft_consensus.cc:729] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.658687 4478 consensus_queue.cc:260] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:14.659281 4535 tablet_bootstrap.cc:492] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:14.660590 4535 tablet_bootstrap.cc:654] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.660624 4478 ts_tablet_manager.cc:1398] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:14.663280 4538 tablet_bootstrap.cc:492] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:14.663388 4538 ts_tablet_manager.cc:1367] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 19:57:14.663820 4535 tablet_bootstrap.cc:492] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:14.663895 4535 ts_tablet_manager.cc:1367] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:14.663856 4538 raft_consensus.cc:348] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.663964 4538 raft_consensus.cc:374] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.664018 4538 raft_consensus.cc:729] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.664135 4538 consensus_queue.cc:260] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.664283 4535 raft_consensus.cc:348] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.664374 4535 raft_consensus.cc:374] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.664400 4535 raft_consensus.cc:729] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.664436 4538 ts_tablet_manager.cc:1398] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.664506 4535 consensus_queue.cc:260] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.664616 4538 tablet_bootstrap.cc:492] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:14.664762 4535 ts_tablet_manager.cc:1398] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.664929 4535 tablet_bootstrap.cc:492] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:14.665524 4538 tablet_bootstrap.cc:654] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.665738 4535 tablet_bootstrap.cc:654] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:14.667568 4535 tablet_bootstrap.cc:492] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82: No bootstrap required, opened a new log
I20240501 19:57:14.668792 4535 ts_tablet_manager.cc:1367] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:14.669315 4535 raft_consensus.cc:348] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.669498 4535 raft_consensus.cc:374] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.669750 4535 raft_consensus.cc:729] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.669940 4535 consensus_queue.cc:260] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.670267 4535 ts_tablet_manager.cc:1398] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.670897 4538 tablet_bootstrap.cc:492] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9: No bootstrap required, opened a new log
I20240501 19:57:14.670979 4538 ts_tablet_manager.cc:1367] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:14.671403 4538 raft_consensus.cc:348] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:14.671522 4538 raft_consensus.cc:374] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:14.671569 4538 raft_consensus.cc:729] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:14.671691 4538 consensus_queue.cc:260] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:14.671960 4538 ts_tablet_manager.cc:1398] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:14.707006 24454 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36710:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:57:14.725029 24856 raft_consensus.cc:1261] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:14.725361 24727 raft_consensus.cc:1261] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:14.725759 2539 consensus_queue.cc:1035] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:14.725948 4350 consensus_queue.cc:1035] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:14.729009 24453 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet c7a53f32ce714398ac2bb7e9878e4383 (table rebalancer_dynamic_table [id=176edbc9ab894af8bd65c3416fd97e32]). Expected version 1 got 0
I20240501 19:57:14.730553 4583 tablet.cc:1722] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:14.731735 24447 catalog_manager.cc:5576] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893) does not have the latest schema for tablet c7a53f32ce714398ac2bb7e9878e4383 (table rebalancer_dynamic_table [id=176edbc9ab894af8bd65c3416fd97e32]). Expected version 1 got 0
I20240501 19:57:14.730867 4583 tablet.cc:1620] T c7a53f32ce714398ac2bb7e9878e4383 P 9081292597be4a72a42b216fddcd7971: MemRowSet was empty: no flush needed.
I20240501 19:57:14.732515 24453 catalog_manager.cc:5576] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299) does not have the latest schema for tablet c7a53f32ce714398ac2bb7e9878e4383 (table rebalancer_dynamic_table [id=176edbc9ab894af8bd65c3416fd97e32]). Expected version 1 got 0
I20240501 19:57:14.735170 4581 tablet.cc:1722] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:14.735606 4581 tablet.cc:1620] T c7a53f32ce714398ac2bb7e9878e4383 P bbf7735f82ca4f65a5170ac7909faa47: MemRowSet was empty: no flush needed.
I20240501 19:57:14.735956 4582 tablet.cc:1722] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:14.737301 24453 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 547d05e083494037ac958fdcd041e13f: rebalancer_dynamic_table [id=176edbc9ab894af8bd65c3416fd97e32] alter complete (version 1)
I20240501 19:57:14.737829 4582 tablet.cc:1620] T c7a53f32ce714398ac2bb7e9878e4383 P 9d64cdfaeb2d46b29845160e395fc241: MemRowSet was empty: no flush needed.
I20240501 19:57:14.793886 3286 raft_consensus.cc:482] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:14.794032 3286 raft_consensus.cc:504] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.794457 3286 leader_election.cc:288] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:14.794900 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ffec91a9d34152b7bc71051ae59c43" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:14.795053 24727 raft_consensus.cc:2454] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:14.795055 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ffec91a9d34152b7bc71051ae59c43" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9" is_pre_election: true
I20240501 19:57:14.795173 27162 raft_consensus.cc:2454] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:14.795356 24922 leader_election.cc:302] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:14.795511 3286 raft_consensus.cc:2781] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:14.795555 3286 raft_consensus.cc:482] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:14.795595 3286 raft_consensus.cc:3037] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.796779 3286 raft_consensus.cc:504] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.797117 3286 leader_election.cc:288] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:14.797534 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ffec91a9d34152b7bc71051ae59c43" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:14.797649 24727 raft_consensus.cc:3037] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.798729 24727 raft_consensus.cc:2454] T 99ffec91a9d34152b7bc71051ae59c43 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:14.799943 24922 leader_election.cc:302] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:14.800338 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ffec91a9d34152b7bc71051ae59c43" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:14.800472 27162 raft_consensus.cc:3037] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.800808 3286 raft_consensus.cc:2781] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:14.800977 3286 raft_consensus.cc:686] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:14.801107 3286 consensus_queue.cc:237] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:14.801615 27162 raft_consensus.cc:2454] T 99ffec91a9d34152b7bc71051ae59c43 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:14.805444 24454 catalog_manager.cc:5484] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:14.936386 3286 raft_consensus.cc:482] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:14.936508 3286 raft_consensus.cc:504] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.936911 3286 leader_election.cc:288] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:14.937307 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be20e5addc5f43c9855df6913d67d17b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
I20240501 19:57:14.937322 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be20e5addc5f43c9855df6913d67d17b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:14.937425 24857 raft_consensus.cc:2454] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:14.937462 24727 raft_consensus.cc:2454] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 0.
I20240501 19:57:14.937728 24922 leader_election.cc:302] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9081292597be4a72a42b216fddcd7971, 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:14.937912 3286 raft_consensus.cc:2781] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:14.937953 3286 raft_consensus.cc:482] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:14.937983 3286 raft_consensus.cc:3037] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.939133 3286 raft_consensus.cc:504] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.939436 3286 leader_election.cc:288] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:14.939677 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be20e5addc5f43c9855df6913d67d17b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:14.939702 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be20e5addc5f43c9855df6913d67d17b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:14.939776 24857 raft_consensus.cc:3037] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.939780 24727 raft_consensus.cc:3037] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:14.940534 24727 raft_consensus.cc:2454] T be20e5addc5f43c9855df6913d67d17b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:14.940538 24857 raft_consensus.cc:2454] T be20e5addc5f43c9855df6913d67d17b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:14.940771 24923 leader_election.cc:302] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:14.940934 3286 raft_consensus.cc:2781] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:14.941020 3286 raft_consensus.cc:686] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:14.941154 3286 consensus_queue.cc:237] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:14.943192 24448 catalog_manager.cc:5484] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.099790 4040 raft_consensus.cc:482] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:15.099901 4040 raft_consensus.cc:504] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:15.100255 4040 leader_election.cc:288] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:15.100524 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0c2b7abe74eb8b8bf30d8b3cf756d" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:15.100531 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0c2b7abe74eb8b8bf30d8b3cf756d" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:15.100633 24725 raft_consensus.cc:2454] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:15.100653 27278 raft_consensus.cc:2454] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 0.
I20240501 19:57:15.100857 27096 leader_election.cc:302] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:15.101027 4040 raft_consensus.cc:2781] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:15.101079 4040 raft_consensus.cc:482] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:15.101131 4040 raft_consensus.cc:3037] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:15.101977 4040 raft_consensus.cc:504] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:15.102303 4040 leader_election.cc:288] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:15.102546 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0c2b7abe74eb8b8bf30d8b3cf756d" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:15.102653 24725 raft_consensus.cc:3037] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:15.102641 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0c2b7abe74eb8b8bf30d8b3cf756d" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:15.102764 27278 raft_consensus.cc:3037] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:15.103595 27278 raft_consensus.cc:2454] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:15.103623 24725 raft_consensus.cc:2454] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 1.
I20240501 19:57:15.103869 27096 leader_election.cc:302] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:15.104081 4040 raft_consensus.cc:2781] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:15.104233 4040 raft_consensus.cc:686] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:15.104353 4040 consensus_queue.cc:237] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:15.106957 24448 catalog_manager.cc:5484] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 1 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:15.138960 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:15.140143 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:15.142843 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:15.147063 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:15.173779 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:15.193138 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:15.195338 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:15.207062 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:15.289748 24987 consensus_queue.cc:237] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:15.291105 24857 raft_consensus.cc:1261] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.291196 24922 consensus_peers.cc:485] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb536b9da41e453a9e366a3e20f7ecc2. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.291327 24725 raft_consensus.cc:1261] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.291440 4627 consensus_queue.cc:1035] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.291782 4627 consensus_queue.cc:1035] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.293393 3286 raft_consensus.cc:2932] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.294229 24725 raft_consensus.cc:2932] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.295828 24857 raft_consensus.cc:2932] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.297242 24448 catalog_manager.cc:5484] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 2, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.301455 4627 consensus_queue.cc:1035] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:15.306797 3286 consensus_queue.cc:1035] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:15.320412 24725 consensus_queue.cc:237] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:15.321645 27421 raft_consensus.cc:1261] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.321720 27031 raft_consensus.cc:1261] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.322039 24662 consensus_peers.cc:485] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f3a2147d24e4fbf92aa1eb88bfb4881. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.322078 4586 consensus_queue.cc:1035] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.322258 4652 consensus_queue.cc:1035] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.323931 4586 raft_consensus.cc:2932] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:15.324338 27421 raft_consensus.cc:2932] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:15.326911 27031 raft_consensus.cc:2932] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:15.328150 24445 catalog_manager.cc:5484] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index -1 to 2, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:15.353355 24725 consensus_queue.cc:237] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:15.354647 27031 raft_consensus.cc:1261] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.354712 24857 raft_consensus.cc:1261] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.355122 4177 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.355275 4652 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:15.355394 24661 consensus_peers.cc:485] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7959d3c11a6d4c7485c8e030e1d5092b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.357017 4177 raft_consensus.cc:2932] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.358615 24857 raft_consensus.cc:2932] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.358898 27031 raft_consensus.cc:2932] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.360117 24453 catalog_manager.cc:5484] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.381711 24595 consensus_queue.cc:237] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:15.383056 24987 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:15.383073 27421 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:57:15.383121 24531 consensus_peers.cc:485] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 718c7f2f1f074e3c84534b502a4809a2. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.383409 2539 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:15.383602 4690 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:15.388079 4690 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:15.388556 24987 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:15.392231 24448 catalog_manager.cc:5484] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:15.392664 27421 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:15.398957 3286 consensus_queue.cc:1035] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:15.410463 4519 consensus_queue.cc:1035] T be20e5addc5f43c9855df6913d67d17b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:15.418179 24725 consensus_queue.cc:237] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:15.419375 24857 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:15.419386 27278 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:15.420156 4177 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:57:15.420320 24660 consensus_peers.cc:485] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcfa6701a958416ca481e0349071caa0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.420490 4177 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:15.422263 4655 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.422645 27280 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.425040 24857 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.425773 24445 catalog_manager.cc:5484] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.453190 27280 consensus_queue.cc:237] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:15.454394 27162 raft_consensus.cc:1261] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.454458 27031 raft_consensus.cc:1261] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.454805 4018 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:15.454978 27229 consensus_peers.cc:485] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdfc4bc6b3b1475e81c2d2b555fcea69. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.455161 4609 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.456624 4018 raft_consensus.cc:2932] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:15.457000 27162 raft_consensus.cc:2932] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:15.459318 27031 raft_consensus.cc:2932] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:15.459921 24453 catalog_manager.cc:5484] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:15.476883 27031 consensus_queue.cc:237] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
W20240501 19:57:15.478179 26965 consensus_peers.cc:485] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb4b1fb143dc410f8f32b68b369b6147. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.478248 24987 raft_consensus.cc:1261] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.478288 24595 raft_consensus.cc:1261] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.478611 4749 consensus_queue.cc:1035] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.478797 3431 consensus_queue.cc:1035] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.480609 4749 raft_consensus.cc:2932] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.481135 24987 raft_consensus.cc:2932] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.483456 24595 raft_consensus.cc:2932] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.483963 24445 catalog_manager.cc:5484] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.504355 27280 consensus_queue.cc:237] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:15.505754 27162 raft_consensus.cc:1261] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.505771 24987 raft_consensus.cc:1261] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.506367 4732 consensus_queue.cc:1035] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.506727 4732 consensus_queue.cc:1035] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:15.506870 27227 consensus_peers.cc:485] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6753d5ac6144c1ba71bc30474f89be7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.508353 4018 raft_consensus.cc:2932] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.508796 24987 raft_consensus.cc:2932] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.510195 27162 raft_consensus.cc:2932] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.512290 24453 catalog_manager.cc:5484] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.531960 27421 consensus_queue.cc:237] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:15.533171 24857 raft_consensus.cc:1261] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.533188 24595 raft_consensus.cc:1261] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.533322 27359 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d80b9d644fb4160950462cdcc895052. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.533563 3666 consensus_queue.cc:1035] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.533774 4788 consensus_queue.cc:1035] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.536221 3666 raft_consensus.cc:2932] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.536655 24595 raft_consensus.cc:2932] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.540303 24451 catalog_manager.cc:5484] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 2, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.540735 24857 raft_consensus.cc:2932] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:15.558727 27031 consensus_queue.cc:237] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:15.560117 27421 raft_consensus.cc:1261] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.560117 27278 raft_consensus.cc:1261] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.560343 26966 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54e20f102ed94db78eb4e735b843df24. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.560485 4752 consensus_queue.cc:1035] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.560796 3431 consensus_queue.cc:1035] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.562397 4752 raft_consensus.cc:2932] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.562789 27421 raft_consensus.cc:2932] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.565922 24453 catalog_manager.cc:5484] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.566563 27278 raft_consensus.cc:2932] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.585793 4588 consensus_queue.cc:1035] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:15.589994 4040 consensus_queue.cc:1035] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:15.601147 27278 consensus_queue.cc:237] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:15.602699 27423 raft_consensus.cc:1261] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.603149 4018 consensus_queue.cc:1035] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.603433 27031 raft_consensus.cc:1261] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.603868 4732 consensus_queue.cc:1035] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:15.604009 27227 consensus_peers.cc:485] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45cabeeee9ad4ec6830a212e93fd2948. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.606468 4018 raft_consensus.cc:2932] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.606952 27031 raft_consensus.cc:2932] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.609452 27423 raft_consensus.cc:2932] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.612143 24451 catalog_manager.cc:5484] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.639181 24857 consensus_queue.cc:237] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:15.640507 27162 raft_consensus.cc:1261] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.640565 24791 consensus_peers.cc:485] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22f2fc10809948cf82331473f18894b3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.640830 27278 raft_consensus.cc:1261] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.640863 4846 consensus_queue.cc:1035] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.641160 4846 consensus_queue.cc:1035] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.642726 4587 raft_consensus.cc:2932] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.643107 27278 raft_consensus.cc:2932] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.646270 24451 catalog_manager.cc:5484] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.647177 27162 raft_consensus.cc:2932] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.664608 27423 consensus_queue.cc:237] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:15.665949 24857 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:15.666085 24987 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240501 19:57:15.666290 27361 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 433347ce828b4d17a70308c421ab4f1f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.666417 3666 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:15.666603 4788 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:15.668902 4787 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:15.669226 24857 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:15.670905 24987 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:15.672894 24451 catalog_manager.cc:5484] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 6 to 7, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 2 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:15.693504 27423 consensus_queue.cc:237] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:15.695019 27278 raft_consensus.cc:1261] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.695125 27360 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca1d950141e14ca68da6d8b3e8254552. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.695411 27031 raft_consensus.cc:1261] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.695421 4787 consensus_queue.cc:1035] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.697218 4787 consensus_queue.cc:1035] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.699957 4787 raft_consensus.cc:2932] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.701756 27031 raft_consensus.cc:2932] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.705190 24453 catalog_manager.cc:5484] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.706665 27278 raft_consensus.cc:2932] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.722512 24987 consensus_queue.cc:237] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:15.724167 24725 raft_consensus.cc:1261] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.724309 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 733a59323d4d434c89874d1dab8c9b06. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.724503 27278 raft_consensus.cc:1261] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.724663 4903 ts_tablet_manager.cc:902] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:15.724714 3286 consensus_queue.cc:1035] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.725059 3286 consensus_queue.cc:1035] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.725193 4903 tablet_copy_client.cc:308] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:57:15.725595 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet fb536b9da41e453a9e366a3e20f7ecc2 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:52411)
I20240501 19:57:15.725690 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet fb536b9da41e453a9e366a3e20f7ecc2 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:52411: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-fb536b9da41e453a9e366a3e20f7ecc2
I20240501 19:57:15.726764 25009 tablet_copy_source_session.cc:205] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.727450 3286 raft_consensus.cc:2932] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:15.727759 4903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb536b9da41e453a9e366a3e20f7ecc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.728009 27278 raft_consensus.cc:2932] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:15.730336 4903 tablet_copy_client.cc:783] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.730484 4903 tablet_copy_client.cc:647] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.731156 24448 catalog_manager.cc:5484] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:15.732574 24725 raft_consensus.cc:2932] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:15.734884 4903 tablet_copy_client.cc:523] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.736505 4903 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:15.740259 4903 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.740522 4903 tablet_bootstrap.cc:492] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:57:15.740691 4903 ts_tablet_manager.cc:1367] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:15.741080 4903 raft_consensus.cc:348] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:15.741163 4903 raft_consensus.cc:729] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.741258 4903 consensus_queue.cc:260] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:15.741638 4903 ts_tablet_manager.cc:1398] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:15.742177 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-fb536b9da41e453a9e366a3e20f7ecc2 received from {username='slave'} at 127.17.65.5:52411
I20240501 19:57:15.742244 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-fb536b9da41e453a9e366a3e20f7ecc2 on tablet fb536b9da41e453a9e366a3e20f7ecc2 with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:57:15.753545 27031 consensus_queue.cc:237] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
W20240501 19:57:15.755105 26966 consensus_peers.cc:485] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38626dc7814f454e85f243fed9607221. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.756795 24987 raft_consensus.cc:1261] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.756798 27162 raft_consensus.cc:1261] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.757182 4010 consensus_queue.cc:1035] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.757359 3431 consensus_queue.cc:1035] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.759384 4010 raft_consensus.cc:2932] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:15.759703 27163 raft_consensus.cc:2932] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:15.763120 24451 catalog_manager.cc:5484] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:15.764714 24987 raft_consensus.cc:2932] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:15.788012 27278 consensus_queue.cc:237] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:15.789790 27163 raft_consensus.cc:1261] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.789793 27423 raft_consensus.cc:1261] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.790243 3848 consensus_queue.cc:1035] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.790557 3848 consensus_queue.cc:1035] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:15.791462 27227 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99737b8809b345ceb15ca74fc313bf4d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.793169 4947 ts_tablet_manager.cc:902] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:15.793788 4947 tablet_copy_client.cc:308] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:57:15.794245 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet 7959d3c11a6d4c7485c8e030e1d5092b from peer b9fd7c49d12b48a7ae73ab09876a0b82 ({username='slave'} at 127.17.65.4:53707)
I20240501 19:57:15.794373 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet 7959d3c11a6d4c7485c8e030e1d5092b from peer b9fd7c49d12b48a7ae73ab09876a0b82 at {username='slave'} at 127.17.65.4:53707: session id = b9fd7c49d12b48a7ae73ab09876a0b82-7959d3c11a6d4c7485c8e030e1d5092b
I20240501 19:57:15.795433 24746 tablet_copy_source_session.cc:205] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.797384 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7959d3c11a6d4c7485c8e030e1d5092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.799824 3848 raft_consensus.cc:2932] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.800189 27423 raft_consensus.cc:2932] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.804368 27163 raft_consensus.cc:2932] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.806723 24453 catalog_manager.cc:5484] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.810762 4947 tablet_copy_client.cc:783] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.810950 4947 tablet_copy_client.cc:647] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.812599 4947 tablet_copy_client.cc:523] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.814129 4947 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:15.818050 4947 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.818235 4947 tablet_bootstrap.cc:492] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap complete.
I20240501 19:57:15.818369 4947 ts_tablet_manager.cc:1367] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:57:15.818722 4947 raft_consensus.cc:348] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:15.818859 4947 raft_consensus.cc:729] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.818980 4947 consensus_queue.cc:260] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:15.819379 4947 ts_tablet_manager.cc:1398] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:15.820097 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-7959d3c11a6d4c7485c8e030e1d5092b received from {username='slave'} at 127.17.65.4:53707
I20240501 19:57:15.820179 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-7959d3c11a6d4c7485c8e030e1d5092b on tablet 7959d3c11a6d4c7485c8e030e1d5092b with peer b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:15.824573 4968 ts_tablet_manager.cc:902] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:15.825278 4968 tablet_copy_client.cc:308] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:57:15.825899 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 718c7f2f1f074e3c84534b502a4809a2 from peer 29d1a658e2b946e0b412e5fef82db9d9 ({username='slave'} at 127.17.65.6:53099)
I20240501 19:57:15.825997 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 718c7f2f1f074e3c84534b502a4809a2 from peer 29d1a658e2b946e0b412e5fef82db9d9 at {username='slave'} at 127.17.65.6:53099: session id = 29d1a658e2b946e0b412e5fef82db9d9-718c7f2f1f074e3c84534b502a4809a2
I20240501 19:57:15.827014 24616 tablet_copy_source_session.cc:205] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.827273 27280 consensus_queue.cc:237] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:15.829246 27423 raft_consensus.cc:1261] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.829946 3848 consensus_queue.cc:1035] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.830024 4968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718c7f2f1f074e3c84534b502a4809a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.830060 27031 raft_consensus.cc:1261] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.830626 3848 consensus_queue.cc:1035] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:15.831197 27228 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 104d22832ca64b91bf802662ba4ec437. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.833012 4609 raft_consensus.cc:2932] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.833421 4968 tablet_copy_client.cc:783] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.833559 4968 tablet_copy_client.cc:647] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.835109 27423 raft_consensus.cc:2932] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.835857 27031 raft_consensus.cc:2932] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.836552 24448 catalog_manager.cc:5484] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:15.841331 4968 tablet_copy_client.cc:523] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.844278 4968 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap starting.
I20240501 19:57:15.858628 24987 consensus_queue.cc:237] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:15.860105 27163 raft_consensus.cc:1261] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.860316 24922 consensus_peers.cc:485] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4f099923d754eb791bd4f567f4d2774. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.860448 27031 raft_consensus.cc:1261] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.860494 4627 consensus_queue.cc:1035] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.860827 4627 consensus_queue.cc:1035] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.862581 4630 raft_consensus.cc:2932] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.863091 27031 raft_consensus.cc:2932] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.864691 27163 raft_consensus.cc:2932] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.866595 24453 catalog_manager.cc:5484] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:15.869272 4968 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.869566 4968 tablet_bootstrap.cc:492] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: Bootstrap complete.
I20240501 19:57:15.869774 4968 ts_tablet_manager.cc:1367] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240501 19:57:15.870261 4968 raft_consensus.cc:348] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:15.870381 4968 raft_consensus.cc:729] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.870551 4968 consensus_queue.cc:260] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:15.870885 4968 ts_tablet_manager.cc:1398] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:15.871461 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-718c7f2f1f074e3c84534b502a4809a2 received from {username='slave'} at 127.17.65.6:53099
I20240501 19:57:15.871554 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 29d1a658e2b946e0b412e5fef82db9d9-718c7f2f1f074e3c84534b502a4809a2 on tablet 718c7f2f1f074e3c84534b502a4809a2 with peer 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:15.871872 4903 ts_tablet_manager.cc:902] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:15.872371 4903 tablet_copy_client.cc:308] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:57:15.873019 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet fcfa6701a958416ca481e0349071caa0 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:37455)
I20240501 19:57:15.873119 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet fcfa6701a958416ca481e0349071caa0 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:37455: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-fcfa6701a958416ca481e0349071caa0
I20240501 19:57:15.874102 24746 tablet_copy_source_session.cc:205] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.875445 4903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfa6701a958416ca481e0349071caa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.877961 4903 tablet_copy_client.cc:783] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.878152 4903 tablet_copy_client.cc:647] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.879312 4903 tablet_copy_client.cc:523] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.880496 4903 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:15.896014 27278 consensus_queue.cc:237] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:15.897336 27423 raft_consensus.cc:1261] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:15.897441 27031 raft_consensus.cc:1261] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:15.897732 27227 consensus_peers.cc:485] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85c67171da5e4b8bbfce0398461ae902. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:15.897738 4018 consensus_queue.cc:1035] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.898375 3848 consensus_queue.cc:1035] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:15.898772 5016 ts_tablet_manager.cc:902] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:15.899402 5016 tablet_copy_client.cc:308] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:57:15.899861 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet bb4b1fb143dc410f8f32b68b369b6147 from peer 9081292597be4a72a42b216fddcd7971 ({username='slave'} at 127.17.65.2:39401)
I20240501 19:57:15.899952 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet bb4b1fb143dc410f8f32b68b369b6147 from peer 9081292597be4a72a42b216fddcd7971 at {username='slave'} at 127.17.65.2:39401: session id = 9081292597be4a72a42b216fddcd7971-bb4b1fb143dc410f8f32b68b369b6147
I20240501 19:57:15.900544 3848 raft_consensus.cc:2932] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.900849 27052 tablet_copy_source_session.cc:205] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.902855 5016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b1fb143dc410f8f32b68b369b6147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.903968 27423 raft_consensus.cc:2932] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.905232 5016 tablet_copy_client.cc:783] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.905376 5016 tablet_copy_client.cc:647] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.905792 27031 raft_consensus.cc:2932] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.907644 4903 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.907946 4903 tablet_bootstrap.cc:492] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:57:15.908125 4903 ts_tablet_manager.cc:1367] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20240501 19:57:15.908571 4903 raft_consensus.cc:348] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:15.908685 4903 raft_consensus.cc:729] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.908792 5016 tablet_copy_client.cc:523] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.908788 4903 consensus_queue.cc:260] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:15.909071 4903 ts_tablet_manager.cc:1398] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:15.909428 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-fcfa6701a958416ca481e0349071caa0 received from {username='slave'} at 127.17.65.5:37455
I20240501 19:57:15.909490 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-fcfa6701a958416ca481e0349071caa0 on tablet fcfa6701a958416ca481e0349071caa0 with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:57:15.909842 24448 catalog_manager.cc:5484] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:15.910486 5016 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:15.915513 5016 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.915669 5024 ts_tablet_manager.cc:902] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:15.915777 5016 tablet_bootstrap.cc:492] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: Bootstrap complete.
I20240501 19:57:15.915935 5016 ts_tablet_manager.cc:1367] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:57:15.916350 5024 tablet_copy_client.cc:308] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:57:15.916400 5016 raft_consensus.cc:348] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:15.917441 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet c6753d5ac6144c1ba71bc30474f89be7 from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:43607)
I20240501 19:57:15.917205 5016 raft_consensus.cc:729] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.917546 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet c6753d5ac6144c1ba71bc30474f89be7 from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:43607: session id = bbf7735f82ca4f65a5170ac7909faa47-c6753d5ac6144c1ba71bc30474f89be7
I20240501 19:57:15.917639 5016 consensus_queue.cc:260] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:15.917961 5016 ts_tablet_manager.cc:1398] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:15.918377 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session 9081292597be4a72a42b216fddcd7971-bb4b1fb143dc410f8f32b68b369b6147 received from {username='slave'} at 127.17.65.2:39401
I20240501 19:57:15.918447 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session 9081292597be4a72a42b216fddcd7971-bb4b1fb143dc410f8f32b68b369b6147 on tablet bb4b1fb143dc410f8f32b68b369b6147 with peer 9081292597be4a72a42b216fddcd7971
I20240501 19:57:15.918516 27314 tablet_copy_source_session.cc:205] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.919565 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6753d5ac6144c1ba71bc30474f89be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.923126 5024 tablet_copy_client.cc:783] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.923277 5024 tablet_copy_client.cc:647] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.926024 5024 tablet_copy_client.cc:523] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.927373 5024 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:15.930992 5024 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.931282 5024 tablet_bootstrap.cc:492] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:15.931473 5024 ts_tablet_manager.cc:1367] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 19:57:15.931970 5024 raft_consensus.cc:348] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:15.932104 5024 raft_consensus.cc:729] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.932274 5024 consensus_queue.cc:260] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:15.932616 5024 ts_tablet_manager.cc:1398] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:15.933108 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-c6753d5ac6144c1ba71bc30474f89be7 received from {username='slave'} at 127.17.65.1:43607
I20240501 19:57:15.933184 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-c6753d5ac6144c1ba71bc30474f89be7 on tablet c6753d5ac6144c1ba71bc30474f89be7 with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:15.937088 5041 ts_tablet_manager.cc:902] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:15.937674 5041 tablet_copy_client.cc:308] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:57:15.938103 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet cdfc4bc6b3b1475e81c2d2b555fcea69 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:40079)
I20240501 19:57:15.938205 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet cdfc4bc6b3b1475e81c2d2b555fcea69 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:40079: session id = 4a6e54e86a9949e5970b42e8e32e7561-cdfc4bc6b3b1475e81c2d2b555fcea69
I20240501 19:57:15.939261 27314 tablet_copy_source_session.cc:205] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.940176 5041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfc4bc6b3b1475e81c2d2b555fcea69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.942414 5041 tablet_copy_client.cc:783] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.942560 5041 tablet_copy_client.cc:647] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.943670 5041 tablet_copy_client.cc:523] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.944787 5041 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:15.947185 5046 ts_tablet_manager.cc:902] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:15.947675 5046 tablet_copy_client.cc:308] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:57:15.948118 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881 from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:38719)
I20240501 19:57:15.948223 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet 0f3a2147d24e4fbf92aa1eb88bfb4881 from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:38719: session id = 53e34bb09d8e41129a5683079ff63610-0f3a2147d24e4fbf92aa1eb88bfb4881
I20240501 19:57:15.949266 24746 tablet_copy_source_session.cc:205] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.949515 5041 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.949755 5041 tablet_bootstrap.cc:492] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:57:15.949931 5041 ts_tablet_manager.cc:1367] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:57:15.950356 5041 raft_consensus.cc:348] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:15.950508 5041 raft_consensus.cc:729] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.950640 5041 consensus_queue.cc:260] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:15.950950 5041 ts_tablet_manager.cc:1398] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:15.951776 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-cdfc4bc6b3b1475e81c2d2b555fcea69 received from {username='slave'} at 127.17.65.8:40079
I20240501 19:57:15.951844 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-cdfc4bc6b3b1475e81c2d2b555fcea69 on tablet cdfc4bc6b3b1475e81c2d2b555fcea69 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:15.951884 5046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.954563 5046 tablet_copy_client.cc:783] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.954831 5046 tablet_copy_client.cc:647] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.957315 5046 tablet_copy_client.cc:523] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.958405 5046 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:15.960160 4947 ts_tablet_manager.cc:902] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:15.960582 4947 tablet_copy_client.cc:308] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:57:15.961512 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet 54e20f102ed94db78eb4e735b843df24 from peer b9fd7c49d12b48a7ae73ab09876a0b82 ({username='slave'} at 127.17.65.4:45851)
I20240501 19:57:15.961623 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet 54e20f102ed94db78eb4e735b843df24 from peer b9fd7c49d12b48a7ae73ab09876a0b82 at {username='slave'} at 127.17.65.4:45851: session id = b9fd7c49d12b48a7ae73ab09876a0b82-54e20f102ed94db78eb4e735b843df24
I20240501 19:57:15.962102 5046 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.962324 5046 tablet_bootstrap.cc:492] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:57:15.962489 5046 ts_tablet_manager.cc:1367] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:15.962515 27052 tablet_copy_source_session.cc:205] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.962926 5046 raft_consensus.cc:348] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:15.963059 5046 raft_consensus.cc:729] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.963186 5046 consensus_queue.cc:260] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:15.963310 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e20f102ed94db78eb4e735b843df24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.963478 5046 ts_tablet_manager.cc:1398] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:15.963939 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-0f3a2147d24e4fbf92aa1eb88bfb4881 received from {username='slave'} at 127.17.65.7:38719
I20240501 19:57:15.964005 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-0f3a2147d24e4fbf92aa1eb88bfb4881 on tablet 0f3a2147d24e4fbf92aa1eb88bfb4881 with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:57:15.965512 4947 tablet_copy_client.cc:783] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.965668 4947 tablet_copy_client.cc:647] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.967144 4947 tablet_copy_client.cc:523] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.968914 4947 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:15.972697 4947 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.972935 4947 tablet_bootstrap.cc:492] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap complete.
I20240501 19:57:15.973091 4947 ts_tablet_manager.cc:1367] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:15.973522 4947 raft_consensus.cc:348] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:15.973649 4947 raft_consensus.cc:729] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.973766 4947 consensus_queue.cc:260] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:15.974042 4947 ts_tablet_manager.cc:1398] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:15.974504 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-54e20f102ed94db78eb4e735b843df24 received from {username='slave'} at 127.17.65.4:45851
I20240501 19:57:15.974591 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-54e20f102ed94db78eb4e735b843df24 on tablet 54e20f102ed94db78eb4e735b843df24 with peer b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:16.047242 5024 ts_tablet_manager.cc:902] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:16.047856 5024 tablet_copy_client.cc:308] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:57:16.049369 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 22f2fc10809948cf82331473f18894b3 from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:40563)
I20240501 19:57:16.049510 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 22f2fc10809948cf82331473f18894b3 from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:40563: session id = bbf7735f82ca4f65a5170ac7909faa47-22f2fc10809948cf82331473f18894b3
I20240501 19:57:16.050499 24878 tablet_copy_source_session.cc:205] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.052107 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f2fc10809948cf82331473f18894b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.054303 5024 tablet_copy_client.cc:783] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.054484 5024 tablet_copy_client.cc:647] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.055802 5024 tablet_copy_client.cc:523] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.057145 5024 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:16.059767 5016 ts_tablet_manager.cc:902] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:16.060237 5016 tablet_copy_client.cc:308] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:57:16.061188 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet 45cabeeee9ad4ec6830a212e93fd2948 from peer 9081292597be4a72a42b216fddcd7971 ({username='slave'} at 127.17.65.2:41119)
I20240501 19:57:16.061312 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet 45cabeeee9ad4ec6830a212e93fd2948 from peer 9081292597be4a72a42b216fddcd7971 at {username='slave'} at 127.17.65.2:41119: session id = 9081292597be4a72a42b216fddcd7971-45cabeeee9ad4ec6830a212e93fd2948
I20240501 19:57:16.062418 27314 tablet_copy_source_session.cc:205] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.067185 5024 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.067432 5024 tablet_bootstrap.cc:492] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:16.067589 5024 ts_tablet_manager.cc:1367] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20240501 19:57:16.067626 5016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cabeeee9ad4ec6830a212e93fd2948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.068058 5024 raft_consensus.cc:348] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:16.068174 5024 raft_consensus.cc:729] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.068317 5024 consensus_queue.cc:260] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:16.068717 5024 ts_tablet_manager.cc:1398] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:16.069901 5016 tablet_copy_client.cc:783] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.070068 5016 tablet_copy_client.cc:647] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.070369 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-22f2fc10809948cf82331473f18894b3 received from {username='slave'} at 127.17.65.1:40563
I20240501 19:57:16.070461 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-22f2fc10809948cf82331473f18894b3 on tablet 22f2fc10809948cf82331473f18894b3 with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:16.072472 5016 tablet_copy_client.cc:523] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.076915 5016 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:16.081209 5016 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.081435 5016 tablet_bootstrap.cc:492] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: Bootstrap complete.
I20240501 19:57:16.081590 5016 ts_tablet_manager.cc:1367] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:57:16.082047 5016 raft_consensus.cc:348] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:16.082178 5016 raft_consensus.cc:729] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.082341 5016 consensus_queue.cc:260] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:16.083719 5016 ts_tablet_manager.cc:1398] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:16.084251 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session 9081292597be4a72a42b216fddcd7971-45cabeeee9ad4ec6830a212e93fd2948 received from {username='slave'} at 127.17.65.2:41119
I20240501 19:57:16.084342 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session 9081292597be4a72a42b216fddcd7971-45cabeeee9ad4ec6830a212e93fd2948 on tablet 45cabeeee9ad4ec6830a212e93fd2948 with peer 9081292597be4a72a42b216fddcd7971
I20240501 19:57:16.102056 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:16.103801 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:16.104154 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:16.107940 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:16.114992 4903 ts_tablet_manager.cc:902] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:16.115198 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:16.115490 4903 tablet_copy_client.cc:308] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.8:35957
I20240501 19:57:16.120422 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:16.120963 27431 tablet_copy_service.cc:140] P 4a6e54e86a9949e5970b42e8e32e7561: Received BeginTabletCopySession request for tablet 5d80b9d644fb4160950462cdcc895052 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:44917)
I20240501 19:57:16.121055 27431 tablet_copy_service.cc:161] P 4a6e54e86a9949e5970b42e8e32e7561: Beginning new tablet copy session on tablet 5d80b9d644fb4160950462cdcc895052 from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:44917: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-5d80b9d644fb4160950462cdcc895052
I20240501 19:57:16.122056 27431 tablet_copy_source_session.cc:205] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.122906 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:16.125862 4903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d80b9d644fb4160950462cdcc895052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.128355 4903 tablet_copy_client.cc:783] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.128510 4903 tablet_copy_client.cc:647] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.131415 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:16.150421 5041 ts_tablet_manager.cc:902] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:16.150903 5041 tablet_copy_client.cc:308] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:57:16.151366 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet 733a59323d4d434c89874d1dab8c9b06 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:57961)
I20240501 19:57:16.151492 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet 733a59323d4d434c89874d1dab8c9b06 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:57961: session id = 4a6e54e86a9949e5970b42e8e32e7561-733a59323d4d434c89874d1dab8c9b06
I20240501 19:57:16.152503 25009 tablet_copy_source_session.cc:205] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.153541 5041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733a59323d4d434c89874d1dab8c9b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.155779 5041 tablet_copy_client.cc:783] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.155933 5041 tablet_copy_client.cc:647] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.164530 5041 tablet_copy_client.cc:523] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.166215 5041 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:16.169761 5041 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.170080 5041 tablet_bootstrap.cc:492] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:57:16.170274 5041 ts_tablet_manager.cc:1367] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.003s
I20240501 19:57:16.171414 5041 raft_consensus.cc:348] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:16.171609 5041 raft_consensus.cc:729] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.171809 5041 consensus_queue.cc:260] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:16.173758 4903 tablet_copy_client.cc:523] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.173820 5041 ts_tablet_manager.cc:1398] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:16.175136 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-733a59323d4d434c89874d1dab8c9b06 received from {username='slave'} at 127.17.65.8:57961
I20240501 19:57:16.175215 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-733a59323d4d434c89874d1dab8c9b06 on tablet 733a59323d4d434c89874d1dab8c9b06 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:16.178293 4903 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:16.182782 4903 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.183023 4903 tablet_bootstrap.cc:492] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:57:16.183187 4903 ts_tablet_manager.cc:1367] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 19:57:16.183611 4903 raft_consensus.cc:348] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:16.183719 4903 raft_consensus.cc:729] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.183835 4903 consensus_queue.cc:260] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:16.184072 4903 ts_tablet_manager.cc:1398] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:16.186232 27431 tablet_copy_service.cc:342] P 4a6e54e86a9949e5970b42e8e32e7561: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-5d80b9d644fb4160950462cdcc895052 received from {username='slave'} at 127.17.65.5:44917
I20240501 19:57:16.186306 27431 tablet_copy_service.cc:433] P 4a6e54e86a9949e5970b42e8e32e7561: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-5d80b9d644fb4160950462cdcc895052 on tablet 5d80b9d644fb4160950462cdcc895052 with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:57:16.205188 5046 ts_tablet_manager.cc:902] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:16.205667 5046 tablet_copy_client.cc:308] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.8:35957
I20240501 19:57:16.206079 27431 tablet_copy_service.cc:140] P 4a6e54e86a9949e5970b42e8e32e7561: Received BeginTabletCopySession request for tablet 433347ce828b4d17a70308c421ab4f1f from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:37097)
I20240501 19:57:16.206175 27431 tablet_copy_service.cc:161] P 4a6e54e86a9949e5970b42e8e32e7561: Beginning new tablet copy session on tablet 433347ce828b4d17a70308c421ab4f1f from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:37097: session id = 53e34bb09d8e41129a5683079ff63610-433347ce828b4d17a70308c421ab4f1f
I20240501 19:57:16.207235 27431 tablet_copy_source_session.cc:205] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.208201 5046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433347ce828b4d17a70308c421ab4f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.210619 5046 tablet_copy_client.cc:783] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.210808 5046 tablet_copy_client.cc:647] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.212065 5046 tablet_copy_client.cc:523] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.213678 5046 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
I20240501 19:57:16.247175 5046 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.247607 5046 tablet_bootstrap.cc:492] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:57:16.247843 5046 ts_tablet_manager.cc:1367] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20240501 19:57:16.248342 5046 raft_consensus.cc:348] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:16.248488 5046 raft_consensus.cc:729] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.248821 5046 consensus_queue.cc:260] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:16.251771 4947 ts_tablet_manager.cc:902] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: Initiating tablet copy from peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:16.252226 4947 tablet_copy_client.cc:308] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.8:35957
I20240501 19:57:16.250283 5046 ts_tablet_manager.cc:1398] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:16.252872 27431 tablet_copy_service.cc:140] P 4a6e54e86a9949e5970b42e8e32e7561: Received BeginTabletCopySession request for tablet ca1d950141e14ca68da6d8b3e8254552 from peer b9fd7c49d12b48a7ae73ab09876a0b82 ({username='slave'} at 127.17.65.4:38021)
I20240501 19:57:16.252985 27431 tablet_copy_service.cc:161] P 4a6e54e86a9949e5970b42e8e32e7561: Beginning new tablet copy session on tablet ca1d950141e14ca68da6d8b3e8254552 from peer b9fd7c49d12b48a7ae73ab09876a0b82 at {username='slave'} at 127.17.65.4:38021: session id = b9fd7c49d12b48a7ae73ab09876a0b82-ca1d950141e14ca68da6d8b3e8254552
I20240501 19:57:16.253885 27431 tablet_copy_source_session.cc:205] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.254616 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1d950141e14ca68da6d8b3e8254552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.257138 27431 tablet_copy_service.cc:342] P 4a6e54e86a9949e5970b42e8e32e7561: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-433347ce828b4d17a70308c421ab4f1f received from {username='slave'} at 127.17.65.7:37097
I20240501 19:57:16.257220 27431 tablet_copy_service.cc:433] P 4a6e54e86a9949e5970b42e8e32e7561: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-433347ce828b4d17a70308c421ab4f1f on tablet 433347ce828b4d17a70308c421ab4f1f with peer 53e34bb09d8e41129a5683079ff63610
I20240501 19:57:16.257834 4947 tablet_copy_client.cc:783] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.258258 4947 tablet_copy_client.cc:647] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.258935 27163 raft_consensus.cc:1203] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:57:16.259230 4947 tablet_copy_client.cc:523] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.260445 4947 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:16.265209 24987 raft_consensus.cc:1203] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.265805 4947 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.266045 4947 tablet_bootstrap.cc:492] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap complete.
I20240501 19:57:16.266188 4947 ts_tablet_manager.cc:1367] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240501 19:57:16.266752 4947 raft_consensus.cc:348] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:16.266871 4947 raft_consensus.cc:729] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.267326 4947 consensus_queue.cc:260] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:16.267633 4947 ts_tablet_manager.cc:1398] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:16.268559 27431 tablet_copy_service.cc:342] P 4a6e54e86a9949e5970b42e8e32e7561: Request end of tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-ca1d950141e14ca68da6d8b3e8254552 received from {username='slave'} at 127.17.65.4:38021
I20240501 19:57:16.268642 27431 tablet_copy_service.cc:433] P 4a6e54e86a9949e5970b42e8e32e7561: ending tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-ca1d950141e14ca68da6d8b3e8254552 on tablet ca1d950141e14ca68da6d8b3e8254552 with peer b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:16.271972 27031 raft_consensus.cc:1203] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.287751 4947 ts_tablet_manager.cc:902] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:16.288200 4947 tablet_copy_client.cc:308] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:57:16.288604 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet 104d22832ca64b91bf802662ba4ec437 from peer b9fd7c49d12b48a7ae73ab09876a0b82 ({username='slave'} at 127.17.65.4:40055)
I20240501 19:57:16.288705 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet 104d22832ca64b91bf802662ba4ec437 from peer b9fd7c49d12b48a7ae73ab09876a0b82 at {username='slave'} at 127.17.65.4:40055: session id = b9fd7c49d12b48a7ae73ab09876a0b82-104d22832ca64b91bf802662ba4ec437
I20240501 19:57:16.289769 27314 tablet_copy_source_session.cc:205] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.291813 5016 ts_tablet_manager.cc:902] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: Initiating tablet copy from peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:16.292093 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104d22832ca64b91bf802662ba4ec437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.292248 5016 tablet_copy_client.cc:308] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.4:45261
I20240501 19:57:16.293629 25009 tablet_copy_service.cc:140] P b9fd7c49d12b48a7ae73ab09876a0b82: Received BeginTabletCopySession request for tablet f4f099923d754eb791bd4f567f4d2774 from peer 9081292597be4a72a42b216fddcd7971 ({username='slave'} at 127.17.65.2:54941)
I20240501 19:57:16.293740 25009 tablet_copy_service.cc:161] P b9fd7c49d12b48a7ae73ab09876a0b82: Beginning new tablet copy session on tablet f4f099923d754eb791bd4f567f4d2774 from peer 9081292597be4a72a42b216fddcd7971 at {username='slave'} at 127.17.65.2:54941: session id = 9081292597be4a72a42b216fddcd7971-f4f099923d754eb791bd4f567f4d2774
I20240501 19:57:16.294682 25009 tablet_copy_source_session.cc:205] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.295578 4947 tablet_copy_client.cc:783] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.295742 4947 tablet_copy_client.cc:647] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.297925 5016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f099923d754eb791bd4f567f4d2774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.300166 5016 tablet_copy_client.cc:783] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.300302 5016 tablet_copy_client.cc:647] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.304184 5024 ts_tablet_manager.cc:902] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:16.304687 5024 tablet_copy_client.cc:308] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:57:16.305603 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet 85c67171da5e4b8bbfce0398461ae902 from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:43607)
I20240501 19:57:16.305721 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet 85c67171da5e4b8bbfce0398461ae902 from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:43607: session id = bbf7735f82ca4f65a5170ac7909faa47-85c67171da5e4b8bbfce0398461ae902
I20240501 19:57:16.306684 27314 tablet_copy_source_session.cc:205] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.307564 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c67171da5e4b8bbfce0398461ae902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.308894 4947 tablet_copy_client.cc:523] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.309963 5024 tablet_copy_client.cc:783] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.310102 5024 tablet_copy_client.cc:647] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.310233 4947 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:16.312049 5024 tablet_copy_client.cc:523] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.313252 5024 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:16.316112 4947 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.316373 4947 tablet_bootstrap.cc:492] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap complete.
I20240501 19:57:16.316553 4947 ts_tablet_manager.cc:1367] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 19:57:16.317903 4947 raft_consensus.cc:348] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:16.318017 4947 raft_consensus.cc:729] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.318159 4947 consensus_queue.cc:260] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:16.321342 4947 ts_tablet_manager.cc:1398] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:57:16.322640 5024 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.322865 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-104d22832ca64b91bf802662ba4ec437 received from {username='slave'} at 127.17.65.4:40055
I20240501 19:57:16.322947 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-104d22832ca64b91bf802662ba4ec437 on tablet 104d22832ca64b91bf802662ba4ec437 with peer b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:16.323377 5024 tablet_bootstrap.cc:492] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:16.323860 5024 ts_tablet_manager.cc:1367] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240501 19:57:16.324921 5024 raft_consensus.cc:348] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:16.328007 5024 raft_consensus.cc:729] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.329475 5016 tablet_copy_client.cc:523] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.330135 5024 consensus_queue.cc:260] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:16.330590 5024 ts_tablet_manager.cc:1398] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 19:57:16.331058 5016 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:16.334025 27032 raft_consensus.cc:1203] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:57:16.341138 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-85c67171da5e4b8bbfce0398461ae902 received from {username='slave'} at 127.17.65.1:43607
I20240501 19:57:16.341224 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-85c67171da5e4b8bbfce0398461ae902 on tablet 85c67171da5e4b8bbfce0398461ae902 with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:16.343772 5024 ts_tablet_manager.cc:902] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:16.345639 5024 tablet_copy_client.cc:308] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:57:16.346112 24987 raft_consensus.cc:1203] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.346904 5016 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.347185 5016 tablet_bootstrap.cc:492] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: Bootstrap complete.
I20240501 19:57:16.347344 5016 ts_tablet_manager.cc:1367] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20240501 19:57:16.347819 5016 raft_consensus.cc:348] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:16.347929 5016 raft_consensus.cc:729] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.348049 5016 consensus_queue.cc:260] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:16.348392 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet 99737b8809b345ceb15ca74fc313bf4d from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:43607)
I20240501 19:57:16.348497 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet 99737b8809b345ceb15ca74fc313bf4d from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:43607: session id = bbf7735f82ca4f65a5170ac7909faa47-99737b8809b345ceb15ca74fc313bf4d
I20240501 19:57:16.349493 27314 tablet_copy_source_session.cc:205] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.350272 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99737b8809b345ceb15ca74fc313bf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.351253 5016 ts_tablet_manager.cc:1398] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:57:16.351841 25009 tablet_copy_service.cc:342] P b9fd7c49d12b48a7ae73ab09876a0b82: Request end of tablet copy session 9081292597be4a72a42b216fddcd7971-f4f099923d754eb791bd4f567f4d2774 received from {username='slave'} at 127.17.65.2:54941
I20240501 19:57:16.351915 25009 tablet_copy_service.cc:433] P b9fd7c49d12b48a7ae73ab09876a0b82: ending tablet copy session 9081292597be4a72a42b216fddcd7971-f4f099923d754eb791bd4f567f4d2774 on tablet f4f099923d754eb791bd4f567f4d2774 with peer 9081292597be4a72a42b216fddcd7971
I20240501 19:57:16.352509 5024 tablet_copy_client.cc:783] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.352690 5024 tablet_copy_client.cc:647] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.355482 5024 tablet_copy_client.cc:523] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.356585 5205 ts_tablet_manager.cc:902] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:16.356920 5024 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:16.357055 5205 tablet_copy_client.cc:308] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:57:16.357951 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet 38626dc7814f454e85f243fed9607221 from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:40115)
I20240501 19:57:16.358049 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet 38626dc7814f454e85f243fed9607221 from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:40115: session id = 9d64cdfaeb2d46b29845160e395fc241-38626dc7814f454e85f243fed9607221
I20240501 19:57:16.358991 27052 tablet_copy_source_session.cc:205] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:16.361306 5024 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.361555 5024 tablet_bootstrap.cc:492] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:16.361816 5024 ts_tablet_manager.cc:1367] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:57:16.362269 5024 raft_consensus.cc:348] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:16.362447 5024 raft_consensus.cc:729] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.362603 5024 consensus_queue.cc:260] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:16.362671 5205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38626dc7814f454e85f243fed9607221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:16.366552 5024 ts_tablet_manager.cc:1398] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:16.370584 5205 tablet_copy_client.cc:783] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:16.371559 5205 tablet_copy_client.cc:647] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:16.372740 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-99737b8809b345ceb15ca74fc313bf4d received from {username='slave'} at 127.17.65.1:43607
I20240501 19:57:16.372817 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-99737b8809b345ceb15ca74fc313bf4d on tablet 99737b8809b345ceb15ca74fc313bf4d with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:16.379470 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:16.381770 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:16.396243 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:16.396814 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:16.398870 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:16.400619 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:16.405267 5205 tablet_copy_client.cc:523] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:16.408337 5205 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:16.411231 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:16.420751 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:16.431291 24595 raft_consensus.cc:1203] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.448742 24725 raft_consensus.cc:1203] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.455147 5205 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:16.455442 5205 tablet_bootstrap.cc:492] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:57:16.455685 5205 ts_tablet_manager.cc:1367] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240501 19:57:16.456148 5205 raft_consensus.cc:348] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:16.456269 5205 raft_consensus.cc:729] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:16.458281 5205 consensus_queue.cc:260] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:16.460455 5205 ts_tablet_manager.cc:1398] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:16.461809 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-38626dc7814f454e85f243fed9607221 received from {username='slave'} at 127.17.65.3:40115
I20240501 19:57:16.461889 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-38626dc7814f454e85f243fed9607221 on tablet 38626dc7814f454e85f243fed9607221 with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:16.504638 24725 raft_consensus.cc:1203] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.509778 27423 raft_consensus.cc:1203] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.559264 24595 raft_consensus.cc:1203] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.571071 27278 raft_consensus.cc:1203] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.584033 27278 raft_consensus.cc:1203] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.6->[2.7-2.7] Dedup: 2.7->[]
I20240501 19:57:16.634194 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:16.635255 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:16.637701 27421 raft_consensus.cc:1203] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.641304 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:16.643209 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:16.655117 27031 raft_consensus.cc:1203] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.661844 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:16.662848 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:16.662998 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:16.667832 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:16.755898 24857 raft_consensus.cc:1203] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.786823 24987 raft_consensus.cc:1203] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.789937 24595 raft_consensus.cc:1203] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.891150 24594 raft_consensus.cc:1203] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.892005 24725 raft_consensus.cc:1203] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.895784 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:16.899286 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:16.899514 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:16.907130 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:16.913965 24987 raft_consensus.cc:1203] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:16.915968 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:16.921255 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:16.928319 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:16.944278 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:16.993448 27278 consensus_queue.cc:237] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:16.994803 27421 raft_consensus.cc:1261] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:16.994916 24987 raft_consensus.cc:1261] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:16.995450 4732 consensus_queue.cc:1035] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:16.995685 4732 consensus_queue.cc:1035] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:16.997328 4018 raft_consensus.cc:2932] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:16.999189 24987 raft_consensus.cc:2932] T 104d22832ca64b91bf802662ba4ec437 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:17.001351 27421 raft_consensus.cc:2932] T 104d22832ca64b91bf802662ba4ec437 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:17.002954 24451 catalog_manager.cc:5484] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:17.009130 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet 104d22832ca64b91bf802662ba4ec437 with delete_type TABLET_DATA_TOMBSTONED (TS 3b5968cebd0b4294898fdd89b7a5b3a1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:17.010989 5339 tablet_replica.cc:330] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:17.011240 5339 raft_consensus.cc:2229] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:17.011497 5339 raft_consensus.cc:2258] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:17.011965 5339 ts_tablet_manager.cc:1875] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:17.014269 5339 ts_tablet_manager.cc:1888] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:17.014400 5339 log.cc:1196] T 104d22832ca64b91bf802662ba4ec437 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/104d22832ca64b91bf802662ba4ec437
I20240501 19:57:17.014910 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet 104d22832ca64b91bf802662ba4ec437 (table rebalancer_extra_table_54 [id=9e901d8fb7ac42ceaa015fb7ea1b0808]) successfully deleted
I20240501 19:57:17.135601 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:17.141400 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:17.143692 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:17.147923 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:17.159243 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:17.164834 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:17.192034 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:17.194564 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:17.250406 27031 consensus_queue.cc:237] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:17.252000 24987 raft_consensus.cc:1261] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:17.252452 24856 raft_consensus.cc:1261] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:17.252974 4749 consensus_queue.cc:1035] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:17.253134 3431 consensus_queue.cc:1035] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:17.254904 4749 raft_consensus.cc:2932] T 38626dc7814f454e85f243fed9607221 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:17.255431 24856 raft_consensus.cc:2932] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:17.258363 24987 raft_consensus.cc:2932] T 38626dc7814f454e85f243fed9607221 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:17.258867 24446 catalog_manager.cc:5484] T 38626dc7814f454e85f243fed9607221 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:17.262401 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 38626dc7814f454e85f243fed9607221 with delete_type TABLET_DATA_TOMBSTONED (TS 29d1a658e2b946e0b412e5fef82db9d9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:17.264394 5413 tablet_replica.cc:330] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:17.264562 5413 raft_consensus.cc:2229] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:17.264698 5413 raft_consensus.cc:2258] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:17.265177 5413 ts_tablet_manager.cc:1875] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:17.268196 5413 ts_tablet_manager.cc:1888] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:17.268335 5413 log.cc:1196] T 38626dc7814f454e85f243fed9607221 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/38626dc7814f454e85f243fed9607221
I20240501 19:57:17.268926 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 38626dc7814f454e85f243fed9607221 (table rebalancer_extra_table_48 [id=31077624d0334b64bd002848767a8906]) successfully deleted
I20240501 19:57:17.385143 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:17.395992 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:17.400199 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:17.405902 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:17.407187 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:17.412719 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:17.434588 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:17.453629 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:17.508443 24987 consensus_queue.cc:237] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:17.509637 27421 raft_consensus.cc:1261] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:17.510227 5478 consensus_queue.cc:1035] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:17.510381 27278 raft_consensus.cc:1261] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:17.510753 5478 consensus_queue.cc:1035] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:17.511885 4627 raft_consensus.cc:2932] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:17.512254 27421 raft_consensus.cc:2932] T 733a59323d4d434c89874d1dab8c9b06 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:17.515170 27278 raft_consensus.cc:2932] T 733a59323d4d434c89874d1dab8c9b06 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:17.515676 24450 catalog_manager.cc:5484] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 2 to 3, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:17.519287 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 733a59323d4d434c89874d1dab8c9b06 with delete_type TABLET_DATA_TOMBSTONED (TS 9081292597be4a72a42b216fddcd7971 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:17.524087 5482 tablet_replica.cc:330] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:17.524240 5482 raft_consensus.cc:2229] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:17.524372 5482 raft_consensus.cc:2258] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:17.524762 5482 ts_tablet_manager.cc:1875] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:17.527709 5482 ts_tablet_manager.cc:1888] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:17.527808 5482 log.cc:1196] T 733a59323d4d434c89874d1dab8c9b06 P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/733a59323d4d434c89874d1dab8c9b06
I20240501 19:57:17.528389 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 733a59323d4d434c89874d1dab8c9b06 (table rebalancer_extra_table_40 [id=4d93be7517bf45f3ac0687fd5daf5f04]) successfully deleted
I20240501 19:57:17.672616 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:17.674741 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:17.675890 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:17.677402 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:17.679558 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:17.689292 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:17.692220 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:17.707620 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:17.778334 27423 consensus_queue.cc:237] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:17.779544 24987 raft_consensus.cc:1261] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:17.779551 27278 raft_consensus.cc:1261] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:17.779937 5123 consensus_queue.cc:1035] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:17.780251 3666 consensus_queue.cc:1035] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:17.781579 5123 raft_consensus.cc:2932] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:17.783097 27278 raft_consensus.cc:2932] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:17.783396 24987 raft_consensus.cc:2932] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:17.785818 24451 catalog_manager.cc:5484] T ca1d950141e14ca68da6d8b3e8254552 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:17.792073 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1d950141e14ca68da6d8b3e8254552 with delete_type TABLET_DATA_TOMBSTONED (TS 3b5968cebd0b4294898fdd89b7a5b3a1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:17.792433 5554 tablet_replica.cc:330] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:17.792587 5554 raft_consensus.cc:2229] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:17.792791 5554 raft_consensus.cc:2258] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:17.793305 5554 ts_tablet_manager.cc:1875] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:17.795986 5554 ts_tablet_manager.cc:1888] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:17.796051 5554 log.cc:1196] T ca1d950141e14ca68da6d8b3e8254552 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/ca1d950141e14ca68da6d8b3e8254552
I20240501 19:57:17.796509 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet ca1d950141e14ca68da6d8b3e8254552 (table rebalancer_extra_table_22 [id=3ee44b7c78df4b3bb22990374bd84fd0]) successfully deleted
I20240501 19:57:17.920969 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:17.928475 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:17.932315 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:17.932863 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:17.933370 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:17.943578 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:17.944820 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:17.955361 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:18.033792 27424 consensus_queue.cc:237] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:18.034996 24987 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:18.035115 27280 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:18.035560 5553 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:18.035789 5123 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:18.037590 5553 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:18.040068 24987 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:18.041972 24453 catalog_manager.cc:5484] T 433347ce828b4d17a70308c421ab4f1f P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index 7 to 8, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 2 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:18.042104 27280 raft_consensus.cc:2932] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:18.045678 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 433347ce828b4d17a70308c421ab4f1f with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:18.046752 5625 tablet_replica.cc:330] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:18.046950 5625 raft_consensus.cc:2229] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:18.047207 5625 raft_consensus.cc:2258] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:18.047818 5625 ts_tablet_manager.cc:1875] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:18.050417 5625 ts_tablet_manager.cc:1888] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 19:57:18.050508 5625 log.cc:1196] T 433347ce828b4d17a70308c421ab4f1f P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/433347ce828b4d17a70308c421ab4f1f
I20240501 19:57:18.051108 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 433347ce828b4d17a70308c421ab4f1f (table rebalance_test_table_17 [id=5cbd2b1998fa41ae85388accf831149b]) successfully deleted
I20240501 19:57:18.186722 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:18.188529 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:18.192384 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:18.193341 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:18.200940 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:18.202078 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:18.208848 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:18.211292 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:18.283285 24857 consensus_queue.cc:237] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:18.284523 24595 raft_consensus.cc:1261] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:18.284973 27278 raft_consensus.cc:1261] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:18.285020 5695 consensus_queue.cc:1035] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:18.285357 4846 consensus_queue.cc:1035] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:18.287351 27278 raft_consensus.cc:2932] T 22f2fc10809948cf82331473f18894b3 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:18.287547 24595 raft_consensus.cc:2932] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:18.286949 5695 raft_consensus.cc:2932] T 22f2fc10809948cf82331473f18894b3 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:18.290758 24448 catalog_manager.cc:5484] T 22f2fc10809948cf82331473f18894b3 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:18.294641 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f2fc10809948cf82331473f18894b3 with delete_type TABLET_DATA_TOMBSTONED (TS 29d1a658e2b946e0b412e5fef82db9d9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:18.295406 5699 tablet_replica.cc:330] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:18.295586 5699 raft_consensus.cc:2229] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:18.295742 5699 raft_consensus.cc:2258] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:18.296250 5699 ts_tablet_manager.cc:1875] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:18.298588 5699 ts_tablet_manager.cc:1888] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:18.298666 5699 log.cc:1196] T 22f2fc10809948cf82331473f18894b3 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/22f2fc10809948cf82331473f18894b3
I20240501 19:57:18.299248 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet 22f2fc10809948cf82331473f18894b3 (table rebalancer_extra_table_62 [id=48bdd49bff724e7eb6d5a822549d0171]) successfully deleted
I20240501 19:57:18.437932 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:18.446156 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:18.448374 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:18.452131 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:18.459198 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:18.469889 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:18.485489 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:18.485945 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:18.549119 27278 consensus_queue.cc:237] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:18.550319 27031 raft_consensus.cc:1261] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:18.550343 24725 raft_consensus.cc:1261] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:18.550662 4732 consensus_queue.cc:1035] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:18.550853 5752 consensus_queue.cc:1035] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:18.552615 5330 raft_consensus.cc:2932] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:18.553035 24725 raft_consensus.cc:2932] T 45cabeeee9ad4ec6830a212e93fd2948 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:18.553081 27031 raft_consensus.cc:2932] T 45cabeeee9ad4ec6830a212e93fd2948 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:18.556140 24446 catalog_manager.cc:5484] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:18.560846 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cabeeee9ad4ec6830a212e93fd2948 with delete_type TABLET_DATA_TOMBSTONED (TS 4a6e54e86a9949e5970b42e8e32e7561 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:18.561787 5772 tablet_replica.cc:330] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:18.561939 5772 raft_consensus.cc:2229] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:18.562060 5772 raft_consensus.cc:2258] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:18.562525 5772 ts_tablet_manager.cc:1875] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:18.564898 5772 ts_tablet_manager.cc:1888] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:18.564978 5772 log.cc:1196] T 45cabeeee9ad4ec6830a212e93fd2948 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/45cabeeee9ad4ec6830a212e93fd2948
I20240501 19:57:18.565580 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 45cabeeee9ad4ec6830a212e93fd2948 (table rebalancer_extra_table_24 [id=792b7817262f482bb3d6469c2b86c91d]) successfully deleted
I20240501 19:57:18.713970 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:18.714687 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:18.716385 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:18.721724 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:18.725821 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:18.736874 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:18.748450 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:18.761937 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:18.823976 27031 consensus_queue.cc:237] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:18.825136 24987 raft_consensus.cc:1261] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:18.825243 27424 raft_consensus.cc:1261] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:18.825559 4749 consensus_queue.cc:1035] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:18.825718 5814 consensus_queue.cc:1035] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:18.827556 5814 raft_consensus.cc:2932] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:18.828037 27424 raft_consensus.cc:2932] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:18.829106 24987 raft_consensus.cc:2932] T 54e20f102ed94db78eb4e735b843df24 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:18.832077 24451 catalog_manager.cc:5484] T 54e20f102ed94db78eb4e735b843df24 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:18.835530 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e20f102ed94db78eb4e735b843df24 with delete_type TABLET_DATA_TOMBSTONED (TS 53e34bb09d8e41129a5683079ff63610 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:18.836913 5848 tablet_replica.cc:330] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:18.837965 5848 raft_consensus.cc:2229] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:18.838111 5848 raft_consensus.cc:2258] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:18.838590 5848 ts_tablet_manager.cc:1875] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:18.841238 5848 ts_tablet_manager.cc:1888] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:18.841360 5848 log.cc:1196] T 54e20f102ed94db78eb4e735b843df24 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/54e20f102ed94db78eb4e735b843df24
I20240501 19:57:18.842137 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet 54e20f102ed94db78eb4e735b843df24 (table rebalancer_extra_table_36 [id=bc65d4b5e31547c58d1ebd24c7a411cb]) successfully deleted
I20240501 19:57:18.967670 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:18.977890 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:18.978479 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:18.982827 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:18.983799 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:19.002941 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:19.011238 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:19.015260 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:19.078050 27424 consensus_queue.cc:237] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:19.079206 27032 raft_consensus.cc:1261] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:19.079278 24595 raft_consensus.cc:1261] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:19.079669 5123 consensus_queue.cc:1035] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:19.079842 3666 consensus_queue.cc:1035] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:19.081604 5123 raft_consensus.cc:2932] T 5d80b9d644fb4160950462cdcc895052 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:19.082017 24595 raft_consensus.cc:2932] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:19.082042 27031 raft_consensus.cc:2932] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:19.085405 24445 catalog_manager.cc:5484] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:19.089294 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d80b9d644fb4160950462cdcc895052 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:19.089613 5924 tablet_replica.cc:330] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:19.089759 5924 raft_consensus.cc:2229] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:19.089875 5924 raft_consensus.cc:2258] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:19.090319 5924 ts_tablet_manager.cc:1875] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:19.093794 5924 ts_tablet_manager.cc:1888] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:19.093876 5924 log.cc:1196] T 5d80b9d644fb4160950462cdcc895052 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/5d80b9d644fb4160950462cdcc895052
I20240501 19:57:19.094460 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet 5d80b9d644fb4160950462cdcc895052 (table rebalancer_extra_table_42 [id=54566ff7e7c94cd3bea5a23769fb663e]) successfully deleted
I20240501 19:57:19.238899 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:19.240561 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:19.241328 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:19.244621 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:19.251513 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:19.259956 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:19.265205 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:19.265604 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:19.334583 27278 consensus_queue.cc:237] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:19.335983 24987 raft_consensus.cc:1261] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:19.336377 5931 consensus_queue.cc:1035] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:19.336555 24594 raft_consensus.cc:1261] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:19.336923 4732 consensus_queue.cc:1035] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:19.338965 4732 raft_consensus.cc:2932] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:19.341168 24987 raft_consensus.cc:2932] T c6753d5ac6144c1ba71bc30474f89be7 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:19.341218 24594 raft_consensus.cc:2932] T c6753d5ac6144c1ba71bc30474f89be7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:19.342806 24448 catalog_manager.cc:5484] T c6753d5ac6144c1ba71bc30474f89be7 P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:19.346279 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet c6753d5ac6144c1ba71bc30474f89be7 with delete_type TABLET_DATA_TOMBSTONED (TS 29d1a658e2b946e0b412e5fef82db9d9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:19.346647 5993 tablet_replica.cc:330] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:19.347170 5993 raft_consensus.cc:2229] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:19.348724 5993 raft_consensus.cc:2258] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:19.349269 5993 ts_tablet_manager.cc:1875] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:19.351581 5993 ts_tablet_manager.cc:1888] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:19.351648 5993 log.cc:1196] T c6753d5ac6144c1ba71bc30474f89be7 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/c6753d5ac6144c1ba71bc30474f89be7
I20240501 19:57:19.352161 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet c6753d5ac6144c1ba71bc30474f89be7 (table rebalancer_extra_table_46 [id=db78701063da482d832f6567d7b0d8bb]) successfully deleted
I20240501 19:57:19.489634 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:19.490118 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:19.493283 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:19.494122 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:19.497926 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:19.512529 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:19.527197 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:19.547988 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:19.598035 27031 consensus_queue.cc:237] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:19.599383 24594 raft_consensus.cc:1261] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:19.599375 24725 raft_consensus.cc:1261] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:19.599792 4749 consensus_queue.cc:1035] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:19.599965 5899 consensus_queue.cc:1035] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:19.602003 5899 raft_consensus.cc:2932] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:19.603976 24727 raft_consensus.cc:2932] T bb4b1fb143dc410f8f32b68b369b6147 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:19.605180 24594 raft_consensus.cc:2932] T bb4b1fb143dc410f8f32b68b369b6147 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:19.605537 24453 catalog_manager.cc:5484] T bb4b1fb143dc410f8f32b68b369b6147 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index 2 to 3, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) evicted. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:19.609963 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b1fb143dc410f8f32b68b369b6147 with delete_type TABLET_DATA_TOMBSTONED (TS b9fd7c49d12b48a7ae73ab09876a0b82 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48994
I20240501 19:57:19.615238 6069 tablet_replica.cc:330] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: stopping tablet replica
I20240501 19:57:19.615401 6069 raft_consensus.cc:2229] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:19.615542 6069 raft_consensus.cc:2258] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:19.615934 6069 ts_tablet_manager.cc:1875] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:19.618019 6069 ts_tablet_manager.cc:1888] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:19.618083 6069 log.cc:1196] T bb4b1fb143dc410f8f32b68b369b6147 P b9fd7c49d12b48a7ae73ab09876a0b82: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb4b1fb143dc410f8f32b68b369b6147
I20240501 19:57:19.618589 24439 catalog_manager.cc:4830] TS b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): tablet bb4b1fb143dc410f8f32b68b369b6147 (table rebalancer_extra_table_60 [id=be41605d64454052a17f58439f9664f9]) successfully deleted
I20240501 19:57:19.743139 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:19.747395 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:19.748667 27142 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:19.754182 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:19.763180 27267 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:19.763875 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:19.764142 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:19.779532 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:19.845759 27280 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69"
dest_uuid: "53e34bb09d8e41129a5683079ff63610"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55170
I20240501 19:57:19.845944 27280 raft_consensus.cc:595] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:19.941478 5931 raft_consensus.cc:979] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: : Instructing follower 29d1a658e2b946e0b412e5fef82db9d9 to start an election
I20240501 19:57:19.941603 5931 raft_consensus.cc:1067] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Signalling peer 29d1a658e2b946e0b412e5fef82db9d9 to start an election
I20240501 19:57:19.942200 27161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69"
dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
from {username='slave'} at 127.17.65.7:53079
I20240501 19:57:19.942368 27161 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:19.942425 27161 raft_consensus.cc:3037] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:19.944067 27161 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:19.944685 27161 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 2 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:19.945196 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:19.945330 27031 raft_consensus.cc:3037] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:19.945715 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:19.945827 27280 raft_consensus.cc:3032] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:19.945868 27280 raft_consensus.cc:729] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:19.946027 27280 consensus_queue.cc:260] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:19.946208 27280 raft_consensus.cc:3037] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:19.946581 27031 raft_consensus.cc:2454] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 2.
I20240501 19:57:19.947219 27280 raft_consensus.cc:2454] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 2.
I20240501 19:57:19.947614 27098 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1, 53e34bb09d8e41129a5683079ff63610; no voters:
I20240501 19:57:19.947816 5270 raft_consensus.cc:2781] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:19.947993 5270 raft_consensus.cc:686] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:19.948148 5270 consensus_queue.cc:237] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:19.949131 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfc4bc6b3b1475e81c2d2b555fcea69" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:19.949255 27423 raft_consensus.cc:3037] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:19.950320 27423 raft_consensus.cc:2454] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 2.
I20240501 19:57:19.951288 24451 catalog_manager.cc:5484] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 1 to 2, leader changed from 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 2 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:19.970996 27142 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:19.985321 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:19.992039 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:19.999527 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:20.000263 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:20.005195 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:20.035454 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:20.040043 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:20.213824 27142 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:20.225589 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:20.229202 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:20.234098 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:20.235713 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:20.238485 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:20.238492 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:20.250789 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:20.319582 24986 consensus_queue.cc:237] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:20.320829 24725 raft_consensus.cc:1261] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:20.320901 27032 raft_consensus.cc:1261] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:20.321235 6187 consensus_queue.cc:1035] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:20.321424 5974 consensus_queue.cc:1035] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:20.323165 5478 raft_consensus.cc:2932] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:20.325055 24725 raft_consensus.cc:2932] T f4f099923d754eb791bd4f567f4d2774 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:20.326336 27032 raft_consensus.cc:2932] T f4f099923d754eb791bd4f567f4d2774 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:20.326582 24445 catalog_manager.cc:5484] T f4f099923d754eb791bd4f567f4d2774 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 2 to 3, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:20.331880 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f099923d754eb791bd4f567f4d2774 with delete_type TABLET_DATA_TOMBSTONED (TS 29d1a658e2b946e0b412e5fef82db9d9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41298
I20240501 19:57:20.332262 6252 tablet_replica.cc:330] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: stopping tablet replica
I20240501 19:57:20.332430 6252 raft_consensus.cc:2229] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:20.332581 6252 raft_consensus.cc:2258] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:20.333061 6252 ts_tablet_manager.cc:1875] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:20.335353 6252 ts_tablet_manager.cc:1888] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:20.335445 6252 log.cc:1196] T f4f099923d754eb791bd4f567f4d2774 P 29d1a658e2b946e0b412e5fef82db9d9: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/wals/f4f099923d754eb791bd4f567f4d2774
I20240501 19:57:20.335958 24440 catalog_manager.cc:4830] TS 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): tablet f4f099923d754eb791bd4f567f4d2774 (table rebalancer_extra_table_32 [id=54f04654f98149f08331cfb97700b4b1]) successfully deleted
I20240501 19:57:20.457098 27032 raft_consensus.cc:1261] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:20.458045 6163 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:20.474395 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:20.477506 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:20.482542 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:20.484421 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:20.487576 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:20.497228 24966 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:20.498107 24707 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:57:20.524288 27142 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:20.553874 24725 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b"
dest_uuid: "9081292597be4a72a42b216fddcd7971"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51668
I20240501 19:57:20.553998 24725 raft_consensus.cc:595] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:20.555765 27424 raft_consensus.cc:1261] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:20.556140 6163 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:20.560035 27278 raft_consensus.cc:1261] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:20.560407 6163 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:20.639761 5461 raft_consensus.cc:979] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: : Instructing follower b9fd7c49d12b48a7ae73ab09876a0b82 to start an election
I20240501 19:57:20.639899 5461 raft_consensus.cc:1067] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Signalling peer b9fd7c49d12b48a7ae73ab09876a0b82 to start an election
I20240501 19:57:20.640425 24986 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b"
dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
from {username='slave'} at 127.17.65.2:54941
I20240501 19:57:20.640558 24986 raft_consensus.cc:482] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:20.640614 24986 raft_consensus.cc:3037] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:20.641796 24986 raft_consensus.cc:504] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:20.642237 24986 leader_election.cc:288] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: Requested vote from peers 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:20.642935 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9d64cdfaeb2d46b29845160e395fc241"
I20240501 19:57:20.642942 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:20.643087 24857 raft_consensus.cc:3037] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:20.643097 24725 raft_consensus.cc:3032] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:20.643142 24725 raft_consensus.cc:729] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:20.643301 24725 consensus_queue.cc:260] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:20.643529 24725 raft_consensus.cc:3037] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:20.644158 24857 raft_consensus.cc:2454] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 2.
I20240501 19:57:20.644896 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959d3c11a6d4c7485c8e030e1d5092b" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:20.645018 27032 raft_consensus.cc:3037] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:20.646059 27032 raft_consensus.cc:2454] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 2.
I20240501 19:57:20.646329 24922 leader_election.cc:302] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9d64cdfaeb2d46b29845160e395fc241, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:20.646579 5974 raft_consensus.cc:2781] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:20.646672 5974 raft_consensus.cc:686] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:20.646800 5974 consensus_queue.cc:237] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:20.648324 24725 raft_consensus.cc:2454] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 2.
I20240501 19:57:20.649673 24452 catalog_manager.cc:5484] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 1 to 2, leader changed from 9081292597be4a72a42b216fddcd7971 (127.17.65.2) to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 2 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:20.702280 27142 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:20.702492 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:20.704352 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:20.704948 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:20.712263 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:20.717499 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:20.727713 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:20.738842 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:20.921614 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:20.927557 27142 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:20.927541 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:20.928720 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:20.935184 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:20.943190 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:20.946969 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:20.949455 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:21.029179 24725 consensus_queue.cc:237] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:21.030653 27278 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:21.031100 5461 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:21.031209 27032 raft_consensus.cc:1261] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:21.031597 5461 consensus_queue.cc:1035] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:21.032883 5461 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:21.033282 27278 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:21.034188 27032 raft_consensus.cc:2932] T fcfa6701a958416ca481e0349071caa0 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:21.036971 24450 catalog_manager.cc:5484] T fcfa6701a958416ca481e0349071caa0 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 6 to 7, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:21.041389 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfa6701a958416ca481e0349071caa0 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:21.042282 6438 tablet_replica.cc:330] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:21.043615 27032 raft_consensus.cc:1261] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:21.043692 6438 raft_consensus.cc:2229] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:21.043895 6438 raft_consensus.cc:2258] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:21.043994 5974 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:21.045491 6438 ts_tablet_manager.cc:1875] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:21.047834 6438 ts_tablet_manager.cc:1888] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:21.047912 6438 log.cc:1196] T fcfa6701a958416ca481e0349071caa0 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/fcfa6701a958416ca481e0349071caa0
I20240501 19:57:21.048413 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet fcfa6701a958416ca481e0349071caa0 (table rebalance_test_table_1 [id=a8f03c8ee2da4d2ba670f1890884cca3]) successfully deleted
I20240501 19:57:21.140844 24725 raft_consensus.cc:1261] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:21.141239 5478 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:21.151314 24856 raft_consensus.cc:1261] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:21.152027 6482 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:21.167922 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:21.177423 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:21.178196 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:21.194851 27142 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:21.195411 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:21.199534 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:21.208393 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:21.213578 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:21.285913 24725 consensus_queue.cc:237] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:21.287331 27031 raft_consensus.cc:1261] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:21.287331 27278 raft_consensus.cc:1261] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:21.287722 5461 consensus_queue.cc:1035] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:21.287916 6349 consensus_queue.cc:1035] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:21.289705 6432 raft_consensus.cc:2932] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:21.290145 27031 raft_consensus.cc:2932] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:21.290139 27278 raft_consensus.cc:2932] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:21.293577 24450 catalog_manager.cc:5484] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:21.297466 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3a2147d24e4fbf92aa1eb88bfb4881 with delete_type TABLET_DATA_TOMBSTONED (TS 4a6e54e86a9949e5970b42e8e32e7561 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:21.297829 6521 tablet_replica.cc:330] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:21.298038 6521 raft_consensus.cc:2229] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:21.298961 6521 raft_consensus.cc:2258] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:21.299482 6521 ts_tablet_manager.cc:1875] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:21.302037 6521 ts_tablet_manager.cc:1888] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:21.302107 6521 log.cc:1196] T 0f3a2147d24e4fbf92aa1eb88bfb4881 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/0f3a2147d24e4fbf92aa1eb88bfb4881
I20240501 19:57:21.302665 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 0f3a2147d24e4fbf92aa1eb88bfb4881 (table rebalancer_extra_table_64 [id=fd08a724d52142f6975ff1e6f926a949]) successfully deleted
I20240501 19:57:21.456483 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:21.459287 27142 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:21.467738 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:21.475564 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:21.484956 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:21.487443 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:21.494997 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:21.500751 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:21.543901 24595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2"
dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51550
I20240501 19:57:21.544077 24595 raft_consensus.cc:595] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:21.694736 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:21.707566 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:21.711045 6479 raft_consensus.cc:979] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: : Instructing follower 29d1a658e2b946e0b412e5fef82db9d9 to start an election
I20240501 19:57:21.711136 6479 raft_consensus.cc:1067] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Signalling peer 29d1a658e2b946e0b412e5fef82db9d9 to start an election
I20240501 19:57:21.712095 27142 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:21.713141 27163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2"
dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
from {username='slave'} at 127.17.65.1:37587
I20240501 19:57:21.713265 27163 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:21.713318 27163 raft_consensus.cc:3037] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:21.714458 27163 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:21.714886 27163 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 2 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:21.715186 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:21.715709 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:21.715826 27423 raft_consensus.cc:3037] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:21.716822 27423 raft_consensus.cc:2454] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 2.
I20240501 19:57:21.717641 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:21.717779 24987 raft_consensus.cc:3037] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:21.718891 24987 raft_consensus.cc:2454] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 2.
I20240501 19:57:21.719338 27097 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:21.719530 6611 raft_consensus.cc:2781] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:21.719615 6611 raft_consensus.cc:686] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 LEADER]: Becoming Leader. State: Replica: 29d1a658e2b946e0b412e5fef82db9d9, State: Running, Role: LEADER
I20240501 19:57:21.719779 6611 consensus_queue.cc:237] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:21.720834 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718c7f2f1f074e3c84534b502a4809a2" candidate_uuid: "29d1a658e2b946e0b412e5fef82db9d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:21.720952 24595 raft_consensus.cc:3032] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:21.720989 24595 raft_consensus.cc:729] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:21.721127 24595 consensus_queue.cc:260] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:21.721288 24595 raft_consensus.cc:3037] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:21.722476 24595 raft_consensus.cc:2454] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d1a658e2b946e0b412e5fef82db9d9 in term 2.
I20240501 19:57:21.724311 24451 catalog_manager.cc:5484] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: term changed from 1 to 2, leader changed from bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) to 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6). New cstate: current_term: 2 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:21.725885 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:21.728432 24987 raft_consensus.cc:1226] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Rejecting Update request from peer bbf7735f82ca4f65a5170ac7909faa47 for earlier term 1. Current term is 2. Ops: []
I20240501 19:57:21.745908 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:21.746455 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:21.771508 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:22.016759 27142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:22.017592 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:22.021939 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:22.032799 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:22.041785 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:22.042034 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:22.050406 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:22.062922 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:22.111603 24987 consensus_queue.cc:237] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:22.112747 27032 raft_consensus.cc:1261] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:22.112790 24726 raft_consensus.cc:1261] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:22.113077 5478 consensus_queue.cc:1035] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:22.113251 6625 consensus_queue.cc:1035] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:22.115056 6625 raft_consensus.cc:2932] T fb536b9da41e453a9e366a3e20f7ecc2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:22.115475 24726 raft_consensus.cc:2932] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:22.118181 27031 raft_consensus.cc:2932] T fb536b9da41e453a9e366a3e20f7ecc2 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:22.118700 24453 catalog_manager.cc:5484] T fb536b9da41e453a9e366a3e20f7ecc2 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 2 to 3, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) evicted. New cstate: current_term: 1 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:22.122442 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet fb536b9da41e453a9e366a3e20f7ecc2 with delete_type TABLET_DATA_TOMBSTONED (TS 9d64cdfaeb2d46b29845160e395fc241 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46810
I20240501 19:57:22.122740 6697 tablet_replica.cc:330] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: stopping tablet replica
I20240501 19:57:22.122928 6697 raft_consensus.cc:2229] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:22.123147 6697 raft_consensus.cc:2258] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:22.123553 6697 ts_tablet_manager.cc:1875] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:22.125540 6697 ts_tablet_manager.cc:1888] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:22.125638 6697 log.cc:1196] T fb536b9da41e453a9e366a3e20f7ecc2 P 9d64cdfaeb2d46b29845160e395fc241: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/wals/fb536b9da41e453a9e366a3e20f7ecc2
I20240501 19:57:22.126183 24439 catalog_manager.cc:4830] TS 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): tablet fb536b9da41e453a9e366a3e20f7ecc2 (table rebalancer_extra_table_58 [id=af59a4587e6e4b38b4931ba4ef49cba5]) successfully deleted
I20240501 19:57:22.183523 24987 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:22.184019 6314 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:22.229538 27423 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:22.229957 6314 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:22.232821 24594 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:22.233222 6314 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:22.471242 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:22.480507 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:22.485627 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:22.490376 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:22.502863 27142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:22.508824 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:22.516825 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:22.519521 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:22.572921 27280 consensus_queue.cc:237] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:22.574129 24594 raft_consensus.cc:1261] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:22.574157 27031 raft_consensus.cc:1261] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:22.574669 6476 consensus_queue.cc:1035] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:22.574823 6477 consensus_queue.cc:1035] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:22.577018 6476 raft_consensus.cc:2932] T 85c67171da5e4b8bbfce0398461ae902 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:22.577339 27031 raft_consensus.cc:2932] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:22.580054 24594 raft_consensus.cc:2932] T 85c67171da5e4b8bbfce0398461ae902 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:22.581265 24445 catalog_manager.cc:5484] T 85c67171da5e4b8bbfce0398461ae902 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index 2 to 3, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) evicted. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:22.585914 27405 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c67171da5e4b8bbfce0398461ae902 with delete_type TABLET_DATA_TOMBSTONED (TS 4a6e54e86a9949e5970b42e8e32e7561 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54928
I20240501 19:57:22.595204 6787 tablet_replica.cc:330] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: stopping tablet replica
I20240501 19:57:22.595356 6787 raft_consensus.cc:2229] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:22.595546 6787 raft_consensus.cc:2258] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:22.596026 6787 ts_tablet_manager.cc:1875] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:22.598117 6787 ts_tablet_manager.cc:1888] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:22.598193 6787 log.cc:1196] T 85c67171da5e4b8bbfce0398461ae902 P 4a6e54e86a9949e5970b42e8e32e7561: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/wals/85c67171da5e4b8bbfce0398461ae902
I20240501 19:57:22.598728 24440 catalog_manager.cc:4830] TS 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): tablet 85c67171da5e4b8bbfce0398461ae902 (table rebalancer_extra_table_30 [id=6052c221112c4cc7ad7bb78b09b40122]) successfully deleted
I20240501 19:57:22.716392 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:22.716392 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:22.717773 27142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:22.729862 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:22.730857 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:22.747049 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:22.750399 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:22.786875 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:22.824702 27280 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d"
dest_uuid: "53e34bb09d8e41129a5683079ff63610"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55312
I20240501 19:57:22.824859 27280 raft_consensus.cc:595] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:22.948395 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:22.953644 27142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:22.959347 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:22.960424 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:22.961396 24575 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:22.972368 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:22.976655 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:22.980696 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:22.987921 6782 raft_consensus.cc:979] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610: : Instructing follower bbf7735f82ca4f65a5170ac7909faa47 to start an election
I20240501 19:57:22.988005 6782 raft_consensus.cc:1067] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Signalling peer bbf7735f82ca4f65a5170ac7909faa47 to start an election
I20240501 19:57:22.988699 24594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d"
dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
from {username='slave'} at 127.17.65.7:36057
I20240501 19:57:22.988858 24594 raft_consensus.cc:482] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:22.988910 24594 raft_consensus.cc:3037] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:22.990047 24594 raft_consensus.cc:504] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:22.992489 24594 leader_election.cc:288] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:22.993206 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "29d1a658e2b946e0b412e5fef82db9d9"
I20240501 19:57:22.993328 27162 raft_consensus.cc:3037] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:22.994400 27162 raft_consensus.cc:2454] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 2.
I20240501 19:57:22.995131 27424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4a6e54e86a9949e5970b42e8e32e7561"
I20240501 19:57:22.995251 27424 raft_consensus.cc:3037] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:22.996304 27424 raft_consensus.cc:2454] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 2.
I20240501 19:57:22.996691 24531 leader_election.cc:302] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:22.996909 6750 raft_consensus.cc:2781] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:22.997084 6750 raft_consensus.cc:686] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [term 2 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:22.997231 6750 consensus_queue.cc:237] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:22.999850 24449 catalog_manager.cc:5484] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 1 to 2, leader changed from 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 2 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:23.000545 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99737b8809b345ceb15ca74fc313bf4d" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:23.000667 27280 raft_consensus.cc:3032] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:23.000705 27280 raft_consensus.cc:729] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:23.000870 27280 consensus_queue.cc:260] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:23.001070 27280 raft_consensus.cc:3037] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:23.002198 27280 raft_consensus.cc:2454] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 2.
I20240501 19:57:23.267441 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:23.270581 27142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:23.275413 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:23.284708 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:23.289218 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:23.289904 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:23.298023 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:23.324016 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:23.374051 27163 consensus_queue.cc:237] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:23.375358 27423 raft_consensus.cc:1261] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:23.375542 27031 raft_consensus.cc:1261] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:23.375891 6852 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:23.376092 7042 consensus_queue.cc:1035] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:23.377982 6852 raft_consensus.cc:2932] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:23.378540 27031 raft_consensus.cc:2932] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:23.381994 24452 catalog_manager.cc:5484] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index 2 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New cstate: current_term: 2 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:23.383468 27423 raft_consensus.cc:2932] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:23.385051 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfc4bc6b3b1475e81c2d2b555fcea69 with delete_type TABLET_DATA_TOMBSTONED (TS 53e34bb09d8e41129a5683079ff63610 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:23.385682 7047 tablet_replica.cc:330] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:23.385880 7047 raft_consensus.cc:2229] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:23.386051 7047 raft_consensus.cc:2258] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:23.386689 7047 ts_tablet_manager.cc:1875] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:23.389843 7047 ts_tablet_manager.cc:1888] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 19:57:23.389967 7047 log.cc:1196] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/cdfc4bc6b3b1475e81c2d2b555fcea69
I20240501 19:57:23.390950 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet cdfc4bc6b3b1475e81c2d2b555fcea69 (table rebalancer_extra_table_56 [id=d75df42152cc45e2ae0ea8b15cf6654a]) successfully deleted
I20240501 19:57:23.408720 27163 raft_consensus.cc:1261] T 99737b8809b345ceb15ca74fc313bf4d P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:23.409279 6479 consensus_queue.cc:1035] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:23.409755 27280 raft_consensus.cc:1261] T 99737b8809b345ceb15ca74fc313bf4d P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:23.415156 6479 consensus_queue.cc:1035] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:23.428094 27423 raft_consensus.cc:1261] T 99737b8809b345ceb15ca74fc313bf4d P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:23.429613 6889 consensus_queue.cc:1035] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:23.545758 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:23.547405 27142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:23.559190 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:23.559799 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:23.562136 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:23.572376 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:23.579095 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:23.613782 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:23.660952 24986 consensus_queue.cc:237] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:23.662070 27031 raft_consensus.cc:1261] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:23.662176 24857 raft_consensus.cc:1261] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:23.662464 6625 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:23.662631 5478 consensus_queue.cc:1035] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:23.664453 5478 raft_consensus.cc:2932] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:23.664887 24857 raft_consensus.cc:2932] T 7959d3c11a6d4c7485c8e030e1d5092b P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:23.665263 27031 raft_consensus.cc:2932] T 7959d3c11a6d4c7485c8e030e1d5092b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:23.668011 24448 catalog_manager.cc:5484] T 7959d3c11a6d4c7485c8e030e1d5092b P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 2 to 4, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New cstate: current_term: 2 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:23.670790 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 7959d3c11a6d4c7485c8e030e1d5092b with delete_type TABLET_DATA_TOMBSTONED (TS 9081292597be4a72a42b216fddcd7971 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:23.671312 7132 tablet_replica.cc:330] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:23.671504 7132 raft_consensus.cc:2229] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:23.671660 7132 raft_consensus.cc:2258] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:23.672149 7132 ts_tablet_manager.cc:1875] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:23.674854 7132 ts_tablet_manager.cc:1888] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 19:57:23.674984 7132 log.cc:1196] T 7959d3c11a6d4c7485c8e030e1d5092b P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/7959d3c11a6d4c7485c8e030e1d5092b
I20240501 19:57:23.676036 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 7959d3c11a6d4c7485c8e030e1d5092b (table rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]) successfully deleted
I20240501 19:57:23.834465 27142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:23.836197 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:23.840490 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:23.841027 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:23.843256 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:23.848325 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:23.859011 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:23.863835 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:23.929504 27163 consensus_queue.cc:237] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:23.930776 27423 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:23.930873 24986 raft_consensus.cc:1261] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Refusing update from remote peer 29d1a658e2b946e0b412e5fef82db9d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:23.931187 7211 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:23.931375 7046 consensus_queue.cc:1035] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:23.933212 7215 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P 29d1a658e2b946e0b412e5fef82db9d9 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:23.933509 27423 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P 4a6e54e86a9949e5970b42e8e32e7561 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:23.933609 24986 raft_consensus.cc:2932] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:23.937949 24453 catalog_manager.cc:5484] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: config changed from index 6 to 8, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) evicted. New cstate: current_term: 2 leader_uuid: "29d1a658e2b946e0b412e5fef82db9d9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:23.941303 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet 718c7f2f1f074e3c84534b502a4809a2 with delete_type TABLET_DATA_TOMBSTONED (TS bbf7735f82ca4f65a5170ac7909faa47 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34316
I20240501 19:57:23.941668 7216 tablet_replica.cc:330] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: stopping tablet replica
I20240501 19:57:23.941874 7216 raft_consensus.cc:2229] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:23.941998 7216 raft_consensus.cc:2258] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:23.942566 7216 ts_tablet_manager.cc:1875] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:23.944833 7216 ts_tablet_manager.cc:1888] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 19:57:23.944941 7216 log.cc:1196] T 718c7f2f1f074e3c84534b502a4809a2 P bbf7735f82ca4f65a5170ac7909faa47: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/wals/718c7f2f1f074e3c84534b502a4809a2
I20240501 19:57:23.945475 24438 catalog_manager.cc:4830] TS bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): tablet 718c7f2f1f074e3c84534b502a4809a2 (table rebalance_test_table_15 [id=905ca9f1ddc548a7a616611f708b5be9]) successfully deleted
I20240501 19:57:23.968997 24594 consensus_queue.cc:237] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:23.970446 27162 raft_consensus.cc:1261] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:23.970763 24530 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 244a44d25272441c939d6d69644eaf56. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:23.971086 6479 consensus_queue.cc:1035] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:23.971144 24725 raft_consensus.cc:1261] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:23.971486 6479 consensus_queue.cc:1035] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:23.973315 7238 raft_consensus.cc:2932] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:23.973762 24725 raft_consensus.cc:2932] T 244a44d25272441c939d6d69644eaf56 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:23.976002 27162 raft_consensus.cc:2932] T 244a44d25272441c939d6d69644eaf56 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:23.976677 24449 catalog_manager.cc:5484] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.011539 27280 consensus_queue.cc:237] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:24.012971 27423 raft_consensus.cc:1261] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.013047 27032 raft_consensus.cc:1261] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.013263 27228 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea7bbf2e0e5f48ed959195deb548b051. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.013370 6779 consensus_queue.cc:1035] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.013610 7102 consensus_queue.cc:1035] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.016147 6779 raft_consensus.cc:2932] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:24.016562 27424 raft_consensus.cc:2932] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:24.018178 27032 raft_consensus.cc:2932] T ea7bbf2e0e5f48ed959195deb548b051 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:24.020125 24453 catalog_manager.cc:5484] T ea7bbf2e0e5f48ed959195deb548b051 P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:24.042630 27032 consensus_queue.cc:237] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.044252 24725 raft_consensus.cc:1261] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.044306 27162 raft_consensus.cc:1261] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.044366 26966 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a73e21b4daff45a3990876f060497386. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.044790 6947 consensus_queue.cc:1035] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.045380 7276 consensus_queue.cc:1035] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.047267 6947 raft_consensus.cc:2932] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.047662 27163 raft_consensus.cc:2932] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.049948 24725 raft_consensus.cc:2932] T a73e21b4daff45a3990876f060497386 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.052326 24445 catalog_manager.cc:5484] T a73e21b4daff45a3990876f060497386 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.074995 24857 consensus_queue.cc:237] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:24.076660 27163 raft_consensus.cc:1261] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.076745 27032 raft_consensus.cc:1261] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.077064 7295 consensus_queue.cc:1035] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.077260 6718 consensus_queue.cc:1035] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:24.077380 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2724041871e04cfdacb91ba5b4026780. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.079080 6718 raft_consensus.cc:2932] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:24.079486 27163 raft_consensus.cc:2932] T 2724041871e04cfdacb91ba5b4026780 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:24.082693 24451 catalog_manager.cc:5484] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:24.084072 27032 raft_consensus.cc:2932] T 2724041871e04cfdacb91ba5b4026780 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:24.103720 24857 consensus_queue.cc:237] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:24.105198 24986 raft_consensus.cc:1261] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:24.105327 27163 raft_consensus.cc:1261] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:57:24.105357 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): Couldn't send request to peer 53e34bb09d8e41129a5683079ff63610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b692b5b96534e79a32a8b06b92f32d6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.105652 7298 consensus_queue.cc:1035] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:24.105859 7295 consensus_queue.cc:1035] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:24.107759 7298 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:24.108202 27163 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:24.110980 24986 raft_consensus.cc:2932] T 3b692b5b96534e79a32a8b06b92f32d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:24.111213 24453 catalog_manager.cc:5484] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 5 to 6, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:24.133602 27280 consensus_queue.cc:237] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:24.135267 24857 raft_consensus.cc:1261] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.135316 27163 raft_consensus.cc:1261] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.135697 6779 consensus_queue.cc:1035] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.135887 7102 consensus_queue.cc:1035] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:24.136222 27227 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79f6ba04101645aa9b5e894b9db07bf5. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.138103 6779 raft_consensus.cc:2932] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:24.138481 24857 raft_consensus.cc:2932] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:24.140797 27163 raft_consensus.cc:2932] T 79f6ba04101645aa9b5e894b9db07bf5 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:24.141525 24445 catalog_manager.cc:5484] T 79f6ba04101645aa9b5e894b9db07bf5 P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New cstate: current_term: 1 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:24.170699 24594 consensus_queue.cc:237] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:24.172190 24857 raft_consensus.cc:1261] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:57:24.172307 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0064e3e3ae1548bba511bd76fb4ec3e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.172315 27163 raft_consensus.cc:1261] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:24.172528 7238 consensus_queue.cc:1035] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:24.172981 7238 consensus_queue.cc:1035] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:24.174777 6889 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:24.175271 27163 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:24.177491 24857 raft_consensus.cc:2932] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:24.179553 24451 catalog_manager.cc:5484] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index 5 to 6, VOTER 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:24.217480 24725 consensus_queue.cc:237] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:24.219089 27278 raft_consensus.cc:1261] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.219235 24660 consensus_peers.cc:485] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6227a1f28614fbcb1b5a573d125e670. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.219422 27032 raft_consensus.cc:1261] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.219509 7370 consensus_queue.cc:1035] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.219849 6762 consensus_queue.cc:1035] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.221755 7370 raft_consensus.cc:2932] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:24.222151 27032 raft_consensus.cc:2932] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:24.225239 27278 raft_consensus.cc:2932] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:24.225428 24453 catalog_manager.cc:5484] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:24.245143 24857 consensus_queue.cc:237] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:24.246551 24986 raft_consensus.cc:1261] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.246693 24594 raft_consensus.cc:1261] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.246807 24793 consensus_peers.cc:485] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63b6863529174c1ab51bb3cb2827351e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.247108 7295 consensus_queue.cc:1035] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.247288 6718 consensus_queue.cc:1035] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.248944 7295 raft_consensus.cc:2932] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:24.249425 24594 raft_consensus.cc:2932] T 63b6863529174c1ab51bb3cb2827351e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:24.251505 24986 raft_consensus.cc:2932] T 63b6863529174c1ab51bb3cb2827351e P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:24.254173 24445 catalog_manager.cc:5484] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index -1 to 2, VOTER 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:24.273975 24857 consensus_queue.cc:237] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:24.275434 27162 raft_consensus.cc:1261] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.275522 27278 raft_consensus.cc:1261] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.275561 24791 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8020390483c4fd98265cbe7d8012435. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.275877 7298 consensus_queue.cc:1035] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.276046 6718 consensus_queue.cc:1035] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.277740 7298 raft_consensus.cc:2932] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:24.278127 27162 raft_consensus.cc:2932] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:24.279875 27278 raft_consensus.cc:2932] T a8020390483c4fd98265cbe7d8012435 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:24.282224 24453 catalog_manager.cc:5484] T a8020390483c4fd98265cbe7d8012435 P 29d1a658e2b946e0b412e5fef82db9d9 reported cstate change: config changed from index -1 to 2, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:24.301525 27032 consensus_queue.cc:237] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.302847 24594 raft_consensus.cc:1261] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.303303 7276 consensus_queue.cc:1035] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.303428 27423 raft_consensus.cc:1261] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.303658 26966 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5515207fb0134f8da7f58f4cf730fae7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.303866 7276 consensus_queue.cc:1035] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.305440 6947 raft_consensus.cc:2932] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.305853 27423 raft_consensus.cc:2932] T 5515207fb0134f8da7f58f4cf730fae7 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.307264 24594 raft_consensus.cc:2932] T 5515207fb0134f8da7f58f4cf730fae7 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.308727 24449 catalog_manager.cc:5484] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.330757 27032 consensus_queue.cc:237] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:24.332379 27423 raft_consensus.cc:1261] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.332625 27278 raft_consensus.cc:1261] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.332700 26965 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abb4f8ef27a54eea89241362c525553b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.332784 7427 consensus_queue.cc:1035] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.333302 7427 consensus_queue.cc:1035] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.334969 7427 raft_consensus.cc:2932] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:24.335435 27423 raft_consensus.cc:2932] T abb4f8ef27a54eea89241362c525553b P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:24.337894 27278 raft_consensus.cc:2932] T abb4f8ef27a54eea89241362c525553b P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:24.338574 24445 catalog_manager.cc:5484] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: config changed from index -1 to 2, VOTER bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1) added. New cstate: current_term: 1 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:24.360702 27424 consensus_queue.cc:237] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:24.362092 27162 raft_consensus.cc:1261] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.362136 27360 consensus_peers.cc:485] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 -> Peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Couldn't send request to peer b9fd7c49d12b48a7ae73ab09876a0b82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.362110 24725 raft_consensus.cc:1261] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 4a6e54e86a9949e5970b42e8e32e7561: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.362540 6612 consensus_queue.cc:1035] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.362727 7462 consensus_queue.cc:1035] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.364732 7462 raft_consensus.cc:2932] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:24.365079 24725 raft_consensus.cc:2932] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:24.368248 27162 raft_consensus.cc:2932] T b4b629378083406b932bfbd5407cac3a P 29d1a658e2b946e0b412e5fef82db9d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:24.368866 24449 catalog_manager.cc:5484] T b4b629378083406b932bfbd5407cac3a P 4a6e54e86a9949e5970b42e8e32e7561 reported cstate change: config changed from index -1 to 2, VOTER b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4) added. New cstate: current_term: 1 leader_uuid: "4a6e54e86a9949e5970b42e8e32e7561" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:24.394598 24594 consensus_queue.cc:237] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.395924 27278 raft_consensus.cc:1261] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:24.395944 24725 raft_consensus.cc:1261] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:24.396137 24530 consensus_peers.cc:485] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893): Couldn't send request to peer 9d64cdfaeb2d46b29845160e395fc241. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd1df8a61b1c41368727514e30e263a8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.396332 6889 consensus_queue.cc:1035] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.396811 6889 consensus_queue.cc:1035] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:24.405575 6479 raft_consensus.cc:2932] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.406021 24725 raft_consensus.cc:2932] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.410079 24445 catalog_manager.cc:5484] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.415208 27278 raft_consensus.cc:2932] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:24.421495 7483 ts_tablet_manager.cc:902] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:24.422044 7483 tablet_copy_client.cc:308] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:57:24.422554 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet 244a44d25272441c939d6d69644eaf56 from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:37211)
I20240501 19:57:24.422644 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet 244a44d25272441c939d6d69644eaf56 from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:37211: session id = 9d64cdfaeb2d46b29845160e395fc241-244a44d25272441c939d6d69644eaf56
I20240501 19:57:24.423465 24616 tablet_copy_source_session.cc:205] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.424268 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244a44d25272441c939d6d69644eaf56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.426574 7483 tablet_copy_client.cc:783] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.426818 7483 tablet_copy_client.cc:647] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.428148 7483 tablet_copy_client.cc:523] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.429263 7483 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:24.432819 7483 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.433097 7483 tablet_bootstrap.cc:492] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:57:24.433269 7483 ts_tablet_manager.cc:1367] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:57:24.433760 7483 raft_consensus.cc:348] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.433862 7483 raft_consensus.cc:729] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.434015 7483 consensus_queue.cc:260] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.434351 7483 ts_tablet_manager.cc:1398] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:24.437147 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-244a44d25272441c939d6d69644eaf56 received from {username='slave'} at 127.17.65.3:37211
I20240501 19:57:24.437227 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-244a44d25272441c939d6d69644eaf56 on tablet 244a44d25272441c939d6d69644eaf56 with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:24.458720 24856 consensus_queue.cc:237] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:24.460299 24725 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:24.460299 27278 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:57:24.460340 24791 consensus_peers.cc:485] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): Couldn't send request to peer 3b5968cebd0b4294898fdd89b7a5b3a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9acd3c8577694e799d3f67a0f2f6adcc. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:24.460721 6718 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:24.460896 7295 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:24.462605 6718 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:24.463063 24725 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:24.466125 24449 catalog_manager.cc:5484] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:24.467252 27278 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:24.530730 7548 ts_tablet_manager.cc:902] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:24.531376 7548 tablet_copy_client.cc:308] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:57:24.531893 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 2724041871e04cfdacb91ba5b4026780 from peer 4a6e54e86a9949e5970b42e8e32e7561 ({username='slave'} at 127.17.65.8:39423)
I20240501 19:57:24.532002 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 2724041871e04cfdacb91ba5b4026780 from peer 4a6e54e86a9949e5970b42e8e32e7561 at {username='slave'} at 127.17.65.8:39423: session id = 4a6e54e86a9949e5970b42e8e32e7561-2724041871e04cfdacb91ba5b4026780
I20240501 19:57:24.532840 24878 tablet_copy_source_session.cc:205] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.534070 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2724041871e04cfdacb91ba5b4026780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.536558 7548 tablet_copy_client.cc:783] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.536722 7548 tablet_copy_client.cc:647] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.538066 7548 tablet_copy_client.cc:523] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.539240 7548 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap starting.
I20240501 19:57:24.544909 7548 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.545179 7548 tablet_bootstrap.cc:492] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: Bootstrap complete.
I20240501 19:57:24.545325 7548 ts_tablet_manager.cc:1367] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 19:57:24.545763 7548 raft_consensus.cc:348] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:24.545868 7548 raft_consensus.cc:729] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6e54e86a9949e5970b42e8e32e7561, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.546043 7548 consensus_queue.cc:260] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:24.546327 7548 ts_tablet_manager.cc:1398] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 19:57:24.546793 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-2724041871e04cfdacb91ba5b4026780 received from {username='slave'} at 127.17.65.8:39423
I20240501 19:57:24.546864 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 4a6e54e86a9949e5970b42e8e32e7561-2724041871e04cfdacb91ba5b4026780 on tablet 2724041871e04cfdacb91ba5b4026780 with peer 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:24.597617 7554 ts_tablet_manager.cc:902] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:24.598202 7554 tablet_copy_client.cc:308] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:57:24.598667 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet ea7bbf2e0e5f48ed959195deb548b051 from peer b9fd7c49d12b48a7ae73ab09876a0b82 ({username='slave'} at 127.17.65.4:40055)
I20240501 19:57:24.598776 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet ea7bbf2e0e5f48ed959195deb548b051 from peer b9fd7c49d12b48a7ae73ab09876a0b82 at {username='slave'} at 127.17.65.4:40055: session id = b9fd7c49d12b48a7ae73ab09876a0b82-ea7bbf2e0e5f48ed959195deb548b051
I20240501 19:57:24.599822 27314 tablet_copy_source_session.cc:205] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.600694 7554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7bbf2e0e5f48ed959195deb548b051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.603879 7554 tablet_copy_client.cc:783] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.604035 7554 tablet_copy_client.cc:647] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.608175 7554 tablet_copy_client.cc:523] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.609786 7554 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap starting.
I20240501 19:57:24.613521 7554 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.617797 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:24.620992 7554 tablet_bootstrap.cc:492] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: Bootstrap complete.
I20240501 19:57:24.621173 7554 ts_tablet_manager.cc:1367] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 19:57:24.622112 7554 raft_consensus.cc:348] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:24.622233 7554 raft_consensus.cc:729] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.622349 7554 consensus_queue.cc:260] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:24.622859 7554 ts_tablet_manager.cc:1398] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:24.625083 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-ea7bbf2e0e5f48ed959195deb548b051 received from {username='slave'} at 127.17.65.4:40055
I20240501 19:57:24.625159 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session b9fd7c49d12b48a7ae73ab09876a0b82-ea7bbf2e0e5f48ed959195deb548b051 on tablet ea7bbf2e0e5f48ed959195deb548b051 with peer b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:24.627125 27405 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:24.635689 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:24.638814 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:24.641745 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:24.646224 27142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:24.656808 7483 ts_tablet_manager.cc:902] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:24.657315 7483 tablet_copy_client.cc:308] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:57:24.657752 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet a73e21b4daff45a3990876f060497386 from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:40115)
I20240501 19:57:24.657862 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet a73e21b4daff45a3990876f060497386 from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:40115: session id = 9d64cdfaeb2d46b29845160e395fc241-a73e21b4daff45a3990876f060497386
I20240501 19:57:24.658869 27052 tablet_copy_source_session.cc:205] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.659771 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e21b4daff45a3990876f060497386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.662007 7483 tablet_copy_client.cc:783] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.663070 7483 tablet_copy_client.cc:647] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.667919 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:24.672987 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:24.690896 7483 tablet_copy_client.cc:523] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.692545 7483 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:24.696027 7578 ts_tablet_manager.cc:902] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:24.696864 7578 tablet_copy_client.cc:308] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:57:24.697409 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet a8020390483c4fd98265cbe7d8012435 from peer 9081292597be4a72a42b216fddcd7971 ({username='slave'} at 127.17.65.2:56795)
I20240501 19:57:24.697527 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet a8020390483c4fd98265cbe7d8012435 from peer 9081292597be4a72a42b216fddcd7971 at {username='slave'} at 127.17.65.2:56795: session id = 9081292597be4a72a42b216fddcd7971-a8020390483c4fd98265cbe7d8012435
I20240501 19:57:24.698148 7483 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.698385 7483 tablet_bootstrap.cc:492] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:57:24.698472 24878 tablet_copy_source_session.cc:205] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.700201 7483 ts_tablet_manager.cc:1367] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 19:57:24.700651 7483 raft_consensus.cc:348] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.700795 7483 raft_consensus.cc:729] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.700950 7483 consensus_queue.cc:260] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.701509 7483 ts_tablet_manager.cc:1398] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:24.702001 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-a73e21b4daff45a3990876f060497386 received from {username='slave'} at 127.17.65.3:40115
I20240501 19:57:24.702085 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-a73e21b4daff45a3990876f060497386 on tablet a73e21b4daff45a3990876f060497386 with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:24.703480 7578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8020390483c4fd98265cbe7d8012435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.707593 7583 ts_tablet_manager.cc:902] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:24.708207 7583 tablet_copy_client.cc:308] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:57:24.708587 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 3b692b5b96534e79a32a8b06b92f32d6 from peer 53e34bb09d8e41129a5683079ff63610 ({username='slave'} at 127.17.65.7:44335)
I20240501 19:57:24.708674 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 3b692b5b96534e79a32a8b06b92f32d6 from peer 53e34bb09d8e41129a5683079ff63610 at {username='slave'} at 127.17.65.7:44335: session id = 53e34bb09d8e41129a5683079ff63610-3b692b5b96534e79a32a8b06b92f32d6
I20240501 19:57:24.709467 24878 tablet_copy_source_session.cc:205] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.710464 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b692b5b96534e79a32a8b06b92f32d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.714700 7583 tablet_copy_client.cc:783] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.715131 7578 tablet_copy_client.cc:783] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.715314 7578 tablet_copy_client.cc:647] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.716329 7578 tablet_copy_client.cc:523] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.717443 7578 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
I20240501 19:57:24.721050 7578 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.721385 7578 tablet_bootstrap.cc:492] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: Bootstrap complete.
I20240501 19:57:24.721637 7578 ts_tablet_manager.cc:1367] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:57:24.722085 7578 raft_consensus.cc:348] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:24.722914 7578 raft_consensus.cc:729] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.723151 7578 consensus_queue.cc:260] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:24.723562 7578 ts_tablet_manager.cc:1398] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:24.724071 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 9081292597be4a72a42b216fddcd7971-a8020390483c4fd98265cbe7d8012435 received from {username='slave'} at 127.17.65.2:56795
I20240501 19:57:24.724144 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 9081292597be4a72a42b216fddcd7971-a8020390483c4fd98265cbe7d8012435 on tablet a8020390483c4fd98265cbe7d8012435 with peer 9081292597be4a72a42b216fddcd7971
I20240501 19:57:24.724534 7589 ts_tablet_manager.cc:902] T 63b6863529174c1ab51bb3cb2827351e P 29d1a658e2b946e0b412e5fef82db9d9: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
F20240501 19:57:24.724907 7589 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0064e3e3ae1548bba511bd76fb4ec3e1.queue_time_us, tablet-download-pool-0064e3e3ae1548bba511bd76fb4ec3e1.queue_time_us) (tablet-download-pool-0064e3e3ae1548bba511bd76fb4ec3e1.run_cpu_time_us, tablet-download-pool-0064e3e3ae1548bba511bd76fb4ec3e1.run_cpu_time_us) (tablet-download-pool-0064e3e3ae1548bba511bd76fb4ec3e1.run_wall_time_us, tablet-download-pool-0064e3e3ae1548bba511bd76fb4ec3e1.run_wall_time_us) (tablet-download-pool-0d91e1c792c749e7883bb996d58d99ea.queue_time_us, tablet-download-pool-0d91e1c792c749e7883bb996d58d99ea.queue_time_us) (tablet-download-pool-0d91e1c792c749e7883bb996d58d99ea.run_cpu_time_us, tablet-download-pool-0d91e1c792c749e7883bb996d58d99ea.run_cpu_time_us) (tablet-download-pool-0d91e1c792c749e7883bb996d58d99ea.run_wall_time_us, tablet-download-pool-0d91e1c792c749e7883bb996d58d99ea.run_wall_time_us) (tablet-download-pool-0ef4152fef8749f0994a952095225134.queue_time_us, tablet-download-pool-0ef4152fef8749f0994a952095225134.queue_time_us) (tablet-download-pool-0ef4152fef8749f0994a952095225134.run_cpu_time_us, tablet-download-pool-0ef4152fef8749f0994a952095225134.run_cpu_time_us) (tablet-download-pool-0ef4152fef8749f0994a952095225134.run_wall_time_us, tablet-download-pool-0ef4152fef8749f0994a952095225134.run_wall_time_us) (tablet-download-pool-3b692b5b96534e79a32a8b06b92f32d6.queue_time_us, tablet-download-pool-3b692b5b96534e79a32a8b06b92f32d6.queue_time_us) (tablet-download-pool-3b692b5b96534e79a32a8b06b92f32d6.run_cpu_time_us, tablet-download-pool-3b692b5b96534e79a32a8b06b92f32d6.run_cpu_time_us) (tablet-download-pool-3b692b5b96534e79a32a8b06b92f32d6.run_wall_time_us, tablet-download-pool-3b692b5b96534e79a32a8b06b92f32d6.run_wall_time_us) (tablet-download-pool-53981ad3756d4c54994f7aea973c5cde.queue_time_us, tablet-download-pool-53981ad3756d4c54994f7aea973c5cde.queue_time_us) (tablet-download-pool-53981ad3756d4c54994f7aea973c5cde.run_cpu_time_us, tablet-download-pool-53981ad3756d4c54994f7aea973c5cde.run_cpu_time_us) (tablet-download-pool-53981ad3756d4c54994f7aea973c5cde.run_wall_time_us, tablet-download-pool-53981ad3756d4c54994f7aea973c5cde.run_wall_time_us) (tablet-download-pool-5a37dfc7a4324e2b8c43acb51d07a277.queue_time_us, tablet-download-pool-5a37dfc7a4324e2b8c43acb51d07a277.queue_time_us) (tablet-download-pool-5a37dfc7a4324e2b8c43acb51d07a277.run_cpu_time_us, tablet-download-pool-5a37dfc7a4324e2b8c43acb51d07a277.run_cpu_time_us) (tablet-download-pool-5a37dfc7a4324e2b8c43acb51d07a277.run_wall_time_us, tablet-download-pool-5a37dfc7a4324e2b8c43acb51d07a277.run_wall_time_us) (tablet-download-pool-63b6863529174c1ab51bb3cb2827351e.queue_time_us, tablet-download-pool-63b6863529174c1ab51bb3cb2827351e.queue_time_us) (tablet-download-pool-6d2603a318c34256a9fa28979ae41fec.queue_time_us, tablet-download-pool-6d2603a318c34256a9fa28979ae41fec.queue_time_us) (tablet-download-pool-6d2603a318c34256a9fa28979ae41fec.run_cpu_time_us, tablet-download-pool-6d2603a318c34256a9fa28979ae41fec.run_cpu_time_us) (tablet-download-pool-6d2603a318c34256a9fa28979ae41fec.run_wall_time_us, tablet-download-pool-6d2603a318c34256a9fa28979ae41fec.run_wall_time_us) (tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.queue_time_us, tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.queue_time_us) (tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.run_cpu_time_us, tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.run_cpu_time_us) (tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.run_wall_time_us, tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.run_wall_time_us) (tablet-download-pool-783db67a272147ed8752c5d6dba4456e.queue_time_us, tablet-download-pool-783db67a272147ed8752c5d6dba4456e.queue_time_us) (tablet-download-pool-783db67a272147ed8752c5d6dba4456e.run_cpu_time_us, tablet-download-pool-783db67a272147ed8752c5d6dba4456e.run_cpu_time_us) (tablet-download-pool-783db67a272147ed8752c5d6dba4456e.run_wall_time_us, tablet-download-pool-783db67a272147ed8752c5d6dba4456e.run_wall_time_us) (tablet-download-pool-a4b240b461b548a087fcabdeb938df1e.queue_time_us, tablet-download-pool-a4b240b461b548a087fcabdeb938df1e.queue_time_us) (tablet-download-pool-a4b240b461b548a087fcabdeb938df1e.run_cpu_time_us, tablet-download-pool-a4b240b461b548a087fcabdeb938df1e.run_cpu_time_us) (tablet-download-pool-a4b240b461b548a087fcabdeb938df1e.run_wall_time_us, tablet-download-pool-a4b240b461b548a087fcabdeb938df1e.run_wall_time_us) (tablet-download-pool-cb0bb0d3396a436c878f56d157f30eda.queue_time_us, tablet-download-pool-cb0bb0d3396a436c878f56d157f30eda.queue_time_us) (tablet-download-pool-cb0bb0d3396a436c878f56d157f30eda.run_cpu_time_us, tablet-download-pool-cb0bb0d3396a436c878f56d157f30eda.run_cpu_time_us) (tablet-download-pool-cb0bb0d3396a436c878f56d157f30eda.run_wall_time_us, tablet-download-pool-cb0bb0d3396a436c878f56d157f30eda.run_wall_time_us) (tablet-download-pool-d29130c48f7a435498e854973a090b37.queue_time_us, tablet-download-pool-d29130c48f7a435498e854973a090b37.queue_time_us) (tablet-download-pool-d29130c48f7a435498e854973a090b37.run_cpu_time_us, tablet-download-pool-d29130c48f7a435498e854973a090b37.run_cpu_time_us) (tablet-download-pool-d29130c48f7a435498e854973a090b37.run_wall_time_us, tablet-download-pool-d29130c48f7a435498e854973a090b37.run_wall_time_us) (tablet-download-pool-e75026e7e70548e190f81bf79fb30926.queue_time_us, tablet-download-pool-e75026e7e70548e190f81bf79fb30926.queue_time_us) (tablet-download-pool-e75026e7e70548e190f81bf79fb30926.run_cpu_time_us, tablet-download-pool-e75026e7e70548e190f81bf79fb30926.run_cpu_time_us) (tablet-download-pool-e75026e7e70548e190f81bf79fb30926.run_wall_time_us, tablet-download-pool-e75026e7e70548e190f81bf79fb30926.run_wall_time_us) (tablet-download-pool-f6dbe23d20b34a348938dcc3571020b7.queue_time_us, tablet-download-pool-f6dbe23d20b34a348938dcc3571020b7.queue_time_us) (tablet-download-pool-f6dbe23d20b34a348938dcc3571020b7.run_cpu_time_us, tablet-download-pool-f6dbe23d20b34a348938dcc3571020b7.run_cpu_time_us) (tablet-download-pool-f6dbe23d20b34a348938dcc3571020b7.run_wall_time_us, tablet-download-pool-f6dbe23d20b34a348938dcc3571020b7.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 19:57:24.720978 7583 tablet_copy_client.cc:647] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Starting download of 1 WAL segments...
@ 0x7f21bb424dcd google::LogMessage::Fail() at ??:0
@ 0x7f21bb428b93 google::LogMessage::SendToLog() at ??:0
I20240501 19:57:24.733330 7583 tablet_copy_client.cc:523] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f21bb4247cc google::LogMessage::Flush() at ??:0
I20240501 19:57:24.735157 7583 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: Bootstrap starting.
@ 0x7f21bb425f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f21bc88e044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f21bc872f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f21bc870efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f21bf42377a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f21bf42ca45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f21bf4ad76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f21bf4c14f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f21bf4ac226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 19:57:24.771178 7578 ts_tablet_manager.cc:902] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: Initiating tablet copy from peer 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:24.771595 7578 tablet_copy_client.cc:308] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.7:39797
I20240501 19:57:24.772019 27314 tablet_copy_service.cc:140] P 53e34bb09d8e41129a5683079ff63610: Received BeginTabletCopySession request for tablet 79f6ba04101645aa9b5e894b9db07bf5 from peer 9081292597be4a72a42b216fddcd7971 ({username='slave'} at 127.17.65.2:41119)
I20240501 19:57:24.772116 27314 tablet_copy_service.cc:161] P 53e34bb09d8e41129a5683079ff63610: Beginning new tablet copy session on tablet 79f6ba04101645aa9b5e894b9db07bf5 from peer 9081292597be4a72a42b216fddcd7971 at {username='slave'} at 127.17.65.2:41119: session id = 9081292597be4a72a42b216fddcd7971-79f6ba04101645aa9b5e894b9db07bf5
I20240501 19:57:24.773048 27314 tablet_copy_source_session.cc:205] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.773871 7578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f6ba04101645aa9b5e894b9db07bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.775784 7548 ts_tablet_manager.cc:902] T 0064e3e3ae1548bba511bd76fb4ec3e1 P 4a6e54e86a9949e5970b42e8e32e7561: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
F20240501 19:57:24.775943 7548 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0064e3e3ae1548bba511bd76fb4ec3e1.queue_time_us, tablet-download-pool-0064e3e3ae1548bba511bd76fb4ec3e1.queue_time_us) (tablet-download-pool-2724041871e04cfdacb91ba5b4026780.queue_time_us, tablet-download-pool-2724041871e04cfdacb91ba5b4026780.queue_time_us) (tablet-download-pool-2724041871e04cfdacb91ba5b4026780.run_cpu_time_us, tablet-download-pool-2724041871e04cfdacb91ba5b4026780.run_cpu_time_us) (tablet-download-pool-2724041871e04cfdacb91ba5b4026780.run_wall_time_us, tablet-download-pool-2724041871e04cfdacb91ba5b4026780.run_wall_time_us) (tablet-download-pool-2e30faedce3742e0ba83de21059613d1.queue_time_us, tablet-download-pool-2e30faedce3742e0ba83de21059613d1.queue_time_us) (tablet-download-pool-2e30faedce3742e0ba83de21059613d1.run_cpu_time_us, tablet-download-pool-2e30faedce3742e0ba83de21059613d1.run_cpu_time_us) (tablet-download-pool-2e30faedce3742e0ba83de21059613d1.run_wall_time_us, tablet-download-pool-2e30faedce3742e0ba83de21059613d1.run_wall_time_us) (tablet-download-pool-39e7bc7629944b3da3fe8fb1ecae32b1.queue_time_us, tablet-download-pool-39e7bc7629944b3da3fe8fb1ecae32b1.queue_time_us) (tablet-download-pool-39e7bc7629944b3da3fe8fb1ecae32b1.run_cpu_time_us, tablet-download-pool-39e7bc7629944b3da3fe8fb1ecae32b1.run_cpu_time_us) (tablet-download-pool-39e7bc7629944b3da3fe8fb1ecae32b1.run_wall_time_us, tablet-download-pool-39e7bc7629944b3da3fe8fb1ecae32b1.run_wall_time_us) (tablet-download-pool-3f2d49dcdd8f49d48d8a9bd6d10c332c.queue_time_us, tablet-download-pool-3f2d49dcdd8f49d48d8a9bd6d10c332c.queue_time_us) (tablet-download-pool-3f2d49dcdd8f49d48d8a9bd6d10c332c.run_cpu_time_us, tablet-download-pool-3f2d49dcdd8f49d48d8a9bd6d10c332c.run_cpu_time_us) (tablet-download-pool-3f2d49dcdd8f49d48d8a9bd6d10c332c.run_wall_time_us, tablet-download-pool-3f2d49dcdd8f49d48d8a9bd6d10c332c.run_wall_time_us) (tablet-download-pool-433347ce828b4d17a70308c421ab4f1f.queue_time_us, tablet-download-pool-433347ce828b4d17a70308c421ab4f1f.queue_time_us) (tablet-download-pool-433347ce828b4d17a70308c421ab4f1f.run_cpu_time_us, tablet-download-pool-433347ce828b4d17a70308c421ab4f1f.run_cpu_time_us) (tablet-download-pool-433347ce828b4d17a70308c421ab4f1f.run_wall_time_us, tablet-download-pool-433347ce828b4d17a70308c421ab4f1f.run_wall_time_us) (tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.queue_time_us, tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.queue_time_us) (tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.run_cpu_time_us, tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.run_cpu_time_us) (tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.run_wall_time_us, tablet-download-pool-718c7f2f1f074e3c84534b502a4809a2.run_wall_time_us) (tablet-download-pool-71a4aa3f94bf4a3b8d84cae1e1765ac9.queue_time_us, tablet-download-pool-71a4aa3f94bf4a3b8d84cae1e1765ac9.queue_time_us) (tablet-download-pool-71a4aa3f94bf4a3b8d84cae1e1765ac9.run_cpu_time_us, tablet-download-pool-71a4aa3f94bf4a3b8d84cae1e1765ac9.run_cpu_time_us) (tablet-download-pool-71a4aa3f94bf4a3b8d84cae1e1765ac9.run_wall_time_us, tablet-download-pool-71a4aa3f94bf4a3b8d84cae1e1765ac9.run_wall_time_us) (tablet-download-pool-733a59323d4d434c89874d1dab8c9b06.queue_time_us, tablet-download-pool-733a59323d4d434c89874d1dab8c9b06.queue_time_us) (tablet-download-pool-733a59323d4d434c89874d1dab8c9b06.run_cpu_time_us, tablet-download-pool-733a59323d4d434c89874d1dab8c9b06.run_cpu_time_us) (tablet-download-pool-733a59323d4d434c89874d1dab8c9b06.run_wall_time_us, tablet-download-pool-733a59323d4d434c89874d1dab8c9b06.run_wall_time_us) (tablet-download-pool-92dd4bc514f146bb89083fe8a9315308.queue_time_us, tablet-download-pool-92dd4bc514f146bb89083fe8a9315308.queue_time_us) (tablet-download-pool-92dd4bc514f146bb89083fe8a9315308.run_cpu_time_us, tablet-download-pool-92dd4bc514f146bb89083fe8a9315308.run_cpu_time_us) (tablet-download-pool-92dd4bc514f146bb89083fe8a9315308.run_wall_time_us, tablet-download-pool-92dd4bc514f146bb89083fe8a9315308.run_wall_time_us) (tablet-download-pool-caf81a72d0304b8aa611d4c1eab5ac26.queue_time_us, tablet-download-pool-caf81a72d0304b8aa611d4c1eab5ac26.queue_time_us) (tablet-download-pool-caf81a72d0304b8aa611d4c1eab5ac26.run_cpu_time_us, tablet-download-pool-caf81a72d0304b8aa611d4c1eab5ac26.run_cpu_time_us) (tablet-download-pool-caf81a72d0304b8aa611d4c1eab5ac26.run_wall_time_us, tablet-download-pool-caf81a72d0304b8aa611d4c1eab5ac26.run_wall_time_us) (tablet-download-pool-cdfc4bc6b3b1475e81c2d2b555fcea69.queue_time_us, tablet-download-pool-cdfc4bc6b3b1475e81c2d2b555fcea69.queue_time_us) (tablet-download-pool-cdfc4bc6b3b1475e81c2d2b555fcea69.run_cpu_time_us, tablet-download-pool-cdfc4bc6b3b1475e81c2d2b555fcea69.run_cpu_time_us) (tablet-download-pool-cdfc4bc6b3b1475e81c2d2b555fcea69.run_wall_time_us, tablet-download-pool-cdfc4bc6b3b1475e81c2d2b555fcea69.run_wall_time_us) (tablet-download-pool-cfb037c9bf1a4ed080745df687ac4a89.queue_time_us, tablet-download-pool-cfb037c9bf1a4ed080745df687ac4a89.queue_time_us) (tablet-download-pool-cfb037c9bf1a4ed080745df687ac4a89.run_cpu_time_us, tablet-download-pool-cfb037c9bf1a4ed080745df687ac4a89.run_cpu_time_us) (tablet-download-pool-cfb037c9bf1a4ed080745df687ac4a89.run_wall_time_us, tablet-download-pool-cfb037c9bf1a4ed080745df687ac4a89.run_wall_time_us) (tablet-download-pool-dc23e8b5ad8a4911ae766492fb823806.queue_time_us, tablet-download-pool-dc23e8b5ad8a4911ae766492fb823806.queue_time_us) (tablet-download-pool-dc23e8b5ad8a4911ae766492fb823806.run_cpu_time_us, tablet-download-pool-dc23e8b5ad8a4911ae766492fb823806.run_cpu_time_us) (tablet-download-pool-dc23e8b5ad8a4911ae766492fb823806.run_wall_time_us, tablet-download-pool-dc23e8b5ad8a4911ae766492fb823806.run_wall_time_us) (tablet-download-pool-f89e81297b66402ea09bdb415fb7f385.queue_time_us, tablet-download-pool-f89e81297b66402ea09bdb415fb7f385.queue_time_us) (tablet-download-pool-f89e81297b66402ea09bdb415fb7f385.run_cpu_time_us, tablet-download-pool-f89e81297b66402ea09bdb415fb7f385.run_cpu_time_us) (tablet-download-pool-f89e81297b66402ea09bdb415fb7f385.run_wall_time_us, tablet-download-pool-f89e81297b66402ea09bdb415fb7f385.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 19:57:24.776378 7578 tablet_copy_client.cc:783] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 0 data blocks...
*** Check failure stack trace: ***
I20240501 19:57:24.776543 7578 tablet_copy_client.cc:647] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.776947 7583 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.777242 7583 tablet_bootstrap.cc:492] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: Bootstrap complete.
I20240501 19:57:24.777395 7583 ts_tablet_manager.cc:1367] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.000s
I20240501 19:57:24.777823 7583 raft_consensus.cc:348] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:24.777941 7583 raft_consensus.cc:729] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.778056 7583 consensus_queue.cc:260] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:24.778328 7583 ts_tablet_manager.cc:1398] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:24.778810 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 53e34bb09d8e41129a5683079ff63610-3b692b5b96534e79a32a8b06b92f32d6 received from {username='slave'} at 127.17.65.7:44335
I20240501 19:57:24.778895 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 53e34bb09d8e41129a5683079ff63610-3b692b5b96534e79a32a8b06b92f32d6 on tablet 3b692b5b96534e79a32a8b06b92f32d6 with peer 53e34bb09d8e41129a5683079ff63610
@ 0x7f5d65369dcd google::LogMessage::Fail() at ??:0
@ 0x7f5d6536db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5d653697cc google::LogMessage::Flush() at ??:0
I20240501 19:57:24.791474 7611 ts_tablet_manager.cc:902] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
@ 0x7f21bf4bbc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 19:57:24.791949 7611 tablet_copy_client.cc:308] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
@ 0x557ebaa9275c std::function<>::operator()() at ??:0
I20240501 19:57:24.792418 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet abb4f8ef27a54eea89241362c525553b from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:60025)
I20240501 19:57:24.792510 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet abb4f8ef27a54eea89241362c525553b from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:60025: session id = bbf7735f82ca4f65a5170ac7909faa47-abb4f8ef27a54eea89241362c525553b
I20240501 19:57:24.793447 27052 tablet_copy_source_session.cc:205] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.794138 7483 ts_tablet_manager.cc:902] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:24.794657 7483 tablet_copy_client.cc:308] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.1:43399
I20240501 19:57:24.794771 7611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb4f8ef27a54eea89241362c525553b. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f21bc875e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 19:57:24.795166 24616 tablet_copy_service.cc:140] P bbf7735f82ca4f65a5170ac7909faa47: Received BeginTabletCopySession request for tablet dd1df8a61b1c41368727514e30e263a8 from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:37211)
I20240501 19:57:24.795243 24616 tablet_copy_service.cc:161] P bbf7735f82ca4f65a5170ac7909faa47: Beginning new tablet copy session on tablet dd1df8a61b1c41368727514e30e263a8 from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:37211: session id = 9d64cdfaeb2d46b29845160e395fc241-dd1df8a61b1c41368727514e30e263a8
@ 0x7f5d6536af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 19:57:24.796062 24616 tablet_copy_source_session.cc:205] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.797573 7611 tablet_copy_client.cc:783] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.797736 7611 tablet_copy_client.cc:647] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.798523 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1df8a61b1c41368727514e30e263a8. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f5d667d3044 kudu::TraceMetrics::InternName() at ??:0
I20240501 19:57:24.800617 7483 tablet_copy_client.cc:783] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.800757 7483 tablet_copy_client.cc:647] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
@ 0x7f5d667b7f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 19:57:24.804224 7620 ts_tablet_manager.cc:902] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:24.804721 7620 tablet_copy_client.cc:308] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.2:36299
I20240501 19:57:24.805122 24746 tablet_copy_service.cc:140] P 9081292597be4a72a42b216fddcd7971: Received BeginTabletCopySession request for tablet d6227a1f28614fbcb1b5a573d125e670 from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:43771)
I20240501 19:57:24.805209 24746 tablet_copy_service.cc:161] P 9081292597be4a72a42b216fddcd7971: Beginning new tablet copy session on tablet d6227a1f28614fbcb1b5a573d125e670 from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:43771: session id = bbf7735f82ca4f65a5170ac7909faa47-d6227a1f28614fbcb1b5a573d125e670
I20240501 19:57:24.806084 24746 tablet_copy_source_session.cc:205] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.806846 7620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6227a1f28614fbcb1b5a573d125e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.808833 7620 tablet_copy_client.cc:783] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.808969 7620 tablet_copy_client.cc:647] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
@ 0x7f5d667b5efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 19:57:24.811923 7578 tablet_copy_client.cc:523] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.813449 7578 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: Bootstrap starting.
@ 0x7f21bc8766ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5d6936877a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f21bc87803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 19:57:24.817404 7611 tablet_copy_client.cc:523] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x557ebaa9275c std::function<>::operator()() at ??:0
I20240501 19:57:24.818605 7611 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
@ 0x7f21bc865807 kudu::Thread::SuperviseThread() at ??:0
I20240501 19:57:24.822039 7620 tablet_copy_client.cc:523] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f21bd0a36db start_thread at ??:0
I20240501 19:57:24.823820 7620 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
I20240501 19:57:24.824955 7483 tablet_copy_client.cc:523] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.826313 7483 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
@ 0x7f21ba5db71f clone at ??:0
I20240501 19:57:24.832051 7483 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.832230 7611 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.832294 7483 tablet_bootstrap.cc:492] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:57:24.832451 7611 tablet_bootstrap.cc:492] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:24.832463 7483 ts_tablet_manager.cc:1367] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 19:57:24.832588 7611 ts_tablet_manager.cc:1367] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 19:57:24.832895 7483 raft_consensus.cc:348] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.833005 7483 raft_consensus.cc:729] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.832984 7611 raft_consensus.cc:348] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:24.833097 7611 raft_consensus.cc:729] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.833153 7483 consensus_queue.cc:260] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.833206 7611 consensus_queue.cc:260] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:24.833432 7483 ts_tablet_manager.cc:1398] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:24.833485 7611 ts_tablet_manager.cc:1398] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:24.834151 24616 tablet_copy_service.cc:342] P bbf7735f82ca4f65a5170ac7909faa47: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-dd1df8a61b1c41368727514e30e263a8 received from {username='slave'} at 127.17.65.3:37211
I20240501 19:57:24.834219 24616 tablet_copy_service.cc:433] P bbf7735f82ca4f65a5170ac7909faa47: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-dd1df8a61b1c41368727514e30e263a8 on tablet dd1df8a61b1c41368727514e30e263a8 with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:24.834266 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-abb4f8ef27a54eea89241362c525553b received from {username='slave'} at 127.17.65.1:60025
I20240501 19:57:24.834337 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-abb4f8ef27a54eea89241362c525553b on tablet abb4f8ef27a54eea89241362c525553b with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:24.838936 7578 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.839219 7578 tablet_bootstrap.cc:492] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: Bootstrap complete.
I20240501 19:57:24.839365 7578 ts_tablet_manager.cc:1367] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.004s
I20240501 19:57:24.839957 7578 raft_consensus.cc:348] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:24.840075 7578 raft_consensus.cc:729] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.840214 7578 consensus_queue.cc:260] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:24.840507 7578 ts_tablet_manager.cc:1398] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:24.843595 27314 tablet_copy_service.cc:342] P 53e34bb09d8e41129a5683079ff63610: Request end of tablet copy session 9081292597be4a72a42b216fddcd7971-79f6ba04101645aa9b5e894b9db07bf5 received from {username='slave'} at 127.17.65.2:41119
I20240501 19:57:24.843670 27314 tablet_copy_service.cc:433] P 53e34bb09d8e41129a5683079ff63610: ending tablet copy session 9081292597be4a72a42b216fddcd7971-79f6ba04101645aa9b5e894b9db07bf5 on tablet 79f6ba04101645aa9b5e894b9db07bf5 with peer 9081292597be4a72a42b216fddcd7971
I20240501 19:57:24.845223 7636 ts_tablet_manager.cc:902] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: Initiating tablet copy from peer 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:24.846230 7636 tablet_copy_client.cc:308] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.3:38893
I20240501 19:57:24.851130 24878 tablet_copy_service.cc:140] P 9d64cdfaeb2d46b29845160e395fc241: Received BeginTabletCopySession request for tablet 9acd3c8577694e799d3f67a0f2f6adcc from peer 3b5968cebd0b4294898fdd89b7a5b3a1 ({username='slave'} at 127.17.65.5:53857)
I20240501 19:57:24.851253 24878 tablet_copy_service.cc:161] P 9d64cdfaeb2d46b29845160e395fc241: Beginning new tablet copy session on tablet 9acd3c8577694e799d3f67a0f2f6adcc from peer 3b5968cebd0b4294898fdd89b7a5b3a1 at {username='slave'} at 127.17.65.5:53857: session id = 3b5968cebd0b4294898fdd89b7a5b3a1-9acd3c8577694e799d3f67a0f2f6adcc
@ 0x7f5d69371a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 19:57:24.852387 24878 tablet_copy_source_session.cc:205] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.853539 7636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acd3c8577694e799d3f67a0f2f6adcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.856225 7636 tablet_copy_client.cc:783] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.856365 7636 tablet_copy_client.cc:647] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.856655 7483 ts_tablet_manager.cc:902] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:24.856971 7620 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.857090 7483 tablet_copy_client.cc:308] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:57:24.857230 7620 tablet_bootstrap.cc:492] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:24.857407 7620 ts_tablet_manager.cc:1367] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240501 19:57:24.857877 7620 raft_consensus.cc:348] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:24.857980 7620 raft_consensus.cc:729] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.858084 7620 consensus_queue.cc:260] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:24.858328 7620 ts_tablet_manager.cc:1398] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:24.858533 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet 5515207fb0134f8da7f58f4cf730fae7 from peer 9d64cdfaeb2d46b29845160e395fc241 ({username='slave'} at 127.17.65.3:40115)
I20240501 19:57:24.858609 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet 5515207fb0134f8da7f58f4cf730fae7 from peer 9d64cdfaeb2d46b29845160e395fc241 at {username='slave'} at 127.17.65.3:40115: session id = 9d64cdfaeb2d46b29845160e395fc241-5515207fb0134f8da7f58f4cf730fae7
I20240501 19:57:24.859524 27052 tablet_copy_source_session.cc:205] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:24.859881 24746 tablet_copy_service.cc:342] P 9081292597be4a72a42b216fddcd7971: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-d6227a1f28614fbcb1b5a573d125e670 received from {username='slave'} at 127.17.65.1:43771
I20240501 19:57:24.859975 24746 tablet_copy_service.cc:433] P 9081292597be4a72a42b216fddcd7971: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-d6227a1f28614fbcb1b5a573d125e670 on tablet d6227a1f28614fbcb1b5a573d125e670 with peer bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:24.860634 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5515207fb0134f8da7f58f4cf730fae7. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f5d693f276f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 19:57:24.862891 7483 tablet_copy_client.cc:783] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:24.863023 7483 tablet_copy_client.cc:647] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:24.865453 7636 tablet_copy_client.cc:523] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.870208 7636 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap starting.
I20240501 19:57:24.874924 7483 tablet_copy_client.cc:523] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:24.876586 7483 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap starting.
I20240501 19:57:24.880599 7483 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.880867 7483 tablet_bootstrap.cc:492] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: Bootstrap complete.
I20240501 19:57:24.881002 7483 ts_tablet_manager.cc:1367] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 19:57:24.881856 7483 raft_consensus.cc:348] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.881963 7483 raft_consensus.cc:729] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.882102 7483 consensus_queue.cc:260] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:24.883602 7483 ts_tablet_manager.cc:1398] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:24.884039 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session 9d64cdfaeb2d46b29845160e395fc241-5515207fb0134f8da7f58f4cf730fae7 received from {username='slave'} at 127.17.65.3:40115
I20240501 19:57:24.884114 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session 9d64cdfaeb2d46b29845160e395fc241-5515207fb0134f8da7f58f4cf730fae7 on tablet 5515207fb0134f8da7f58f4cf730fae7 with peer 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:24.906363 7636 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:24.906635 7636 tablet_bootstrap.cc:492] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: Bootstrap complete.
I20240501 19:57:24.906778 7636 ts_tablet_manager.cc:1367] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent bootstrapping tablet: real 0.037s user 0.019s sys 0.007s
I20240501 19:57:24.907256 7636 raft_consensus.cc:348] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:24.907378 7636 raft_consensus.cc:729] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Initialized, Role: FOLLOWER
I20240501 19:57:24.907529 7636 consensus_queue.cc:260] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:24.909561 7636 ts_tablet_manager.cc:1398] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 19:57:24.910032 24878 tablet_copy_service.cc:342] P 9d64cdfaeb2d46b29845160e395fc241: Request end of tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-9acd3c8577694e799d3f67a0f2f6adcc received from {username='slave'} at 127.17.65.5:53857
I20240501 19:57:24.910118 24878 tablet_copy_service.cc:433] P 9d64cdfaeb2d46b29845160e395fc241: ending tablet copy session 3b5968cebd0b4294898fdd89b7a5b3a1-9acd3c8577694e799d3f67a0f2f6adcc on tablet 9acd3c8577694e799d3f67a0f2f6adcc with peer 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:57:24.912380 27424 raft_consensus.cc:1203] T 2724041871e04cfdacb91ba5b4026780 P 4a6e54e86a9949e5970b42e8e32e7561 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7f5d694064f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f5d693f1226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f5d69400c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ef43d4375c std::function<>::operator()() at ??:0
@ 0x7f5d667bae06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5d667bb6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5d667bd03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ef43d4375c std::function<>::operator()() at ??:0
@ 0x7f5d667aa807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5d66fe86db start_thread at ??:0
@ 0x7f5d6452071f clone at ??:0
I20240501 19:57:25.021746 24858 raft_consensus.cc:1203] T 244a44d25272441c939d6d69644eaf56 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:25.038986 24986 raft_consensus.cc:1203] T ea7bbf2e0e5f48ed959195deb548b051 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:25.062135 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:25.062973 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:25.076051 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:25.096155 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:25.109292 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:25.120559 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:25.149922 24858 raft_consensus.cc:1203] T a73e21b4daff45a3990876f060497386 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:25.180106 24595 raft_consensus.cc:1203] T abb4f8ef27a54eea89241362c525553b P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:25.212173 24725 raft_consensus.cc:1203] T a8020390483c4fd98265cbe7d8012435 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:25.220079 27280 raft_consensus.cc:1203] T 3b692b5b96534e79a32a8b06b92f32d6 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240501 19:57:25.250808 27229 connection.cc:535] client connection to 127.17.65.6:44579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.250959 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.251680 24662 connection.cc:535] client connection to 127.17.65.6:44579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.251799 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.252032 24924 connection.cc:535] client connection to 127.17.65.6:44579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.252076 24531 connection.cc:535] client connection to 127.17.65.6:44579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.252166 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.252159 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.252190 24793 connection.cc:535] client connection to 127.17.65.6:44579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.252277 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.252679 26967 connection.cc:535] client connection to 127.17.65.6:44579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.252806 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.253715 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:25.251715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.65.6:44579 (local address 127.17.65.7:56369)
0501 19:57:25.251930 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:25.251948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:25.252052 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:25.253134 (+ 1082us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
W20240501 19:57:25.253952 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.254019 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.254058 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.254092 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.254122 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.254154 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.255107 7723 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:25.252886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.65.6:44579 (local address 127.17.65.1:33231)
0501 19:57:25.254469 (+ 1583us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:25.254587 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1500,"thread_start_us":50,"threads_started":1}
W20240501 19:57:25.255260 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.255318 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.255350 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.255383 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.255420 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.255450 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256263 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256330 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256372 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256435 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256577 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256655 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256856 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256937 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.256981 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257117 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257194 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257241 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257288 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:25.257316 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257364 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257386 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257421 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257457 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257486 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257500 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257536 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.257582 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.258044 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.258323 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.265470 27229 connection.cc:535] client connection to 127.17.65.8:35957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.265591 26967 connection.cc:535] client connection to 127.17.65.8:35957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.265626 24531 connection.cc:535] client connection to 127.17.65.8:35957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.266059 24793 connection.cc:535] client connection to 127.17.65.8:35957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.266144 24662 connection.cc:535] client connection to 127.17.65.8:35957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.266867 24924 connection.cc:535] client connection to 127.17.65.8:35957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:25.267287 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267352 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267398 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267436 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267489 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267524 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267552 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267544 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267586 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267612 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267880 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.267951 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.268653 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.268718 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.268997 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.269671 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.269737 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:25.269783 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.269829 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.269878 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.275846 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.275918 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.275943 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.282152 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:25.287928 24725 raft_consensus.cc:1203] T 79f6ba04101645aa9b5e894b9db07bf5 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:25.294914 24858 raft_consensus.cc:1203] T 5515207fb0134f8da7f58f4cf730fae7 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:25.297684 27031 raft_consensus.cc:1203] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240501 19:57:25.298146 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:25.331321 24858 raft_consensus.cc:1203] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:57:25.333607 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.335906 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.337234 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.339247 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:25.346518 24595 raft_consensus.cc:1203] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:57:25.351100 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.359505 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.378140 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.380086 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.381700 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.393268 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:25.398370 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:25.406133 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:25.408967 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:25.410723 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:25.414067 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:25.419392 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 19:57:25.425714 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.452224 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.463656 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.469189 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:25.473928 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:25.592288 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:25.601848 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:25.617033 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:25.617285 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:25.618045 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:25.625699 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 19:57:25.730525 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:25.807802 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:25.816431 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:25.818815 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:25.822218 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:25.828205 7184 raft_consensus.cc:482] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:25.828317 7184 raft_consensus.cc:504] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:25.828692 7184 leader_election.cc:288] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:25.829010 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0c2b7abe74eb8b8bf30d8b3cf756d" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
W20240501 19:57:25.830181 27229 leader_election.cc:334] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:25.830245 27229 leader_election.cc:302] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 9081292597be4a72a42b216fddcd7971
I20240501 19:57:25.830372 7184 raft_consensus.cc:2726] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:25.841220 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:25.841876 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:25.846518 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:25.887337 6762 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:25.887465 6762 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:25.887888 6762 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:25.889182 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:25.889384 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:25.889473 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:25.889613 6762 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:25.898293 24725 consensus_queue.cc:237] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:25.899495 27280 raft_consensus.cc:1261] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:25.899657 24594 raft_consensus.cc:1261] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:25.899871 7370 consensus_queue.cc:1035] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:25.900153 6762 consensus_queue.cc:1035] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:25.901816 7370 raft_consensus.cc:2932] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:25.902307 24594 raft_consensus.cc:2932] T d6227a1f28614fbcb1b5a573d125e670 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:25.903252 27278 raft_consensus.cc:2932] T d6227a1f28614fbcb1b5a573d125e670 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:25.905409 24451 catalog_manager.cc:5484] T d6227a1f28614fbcb1b5a573d125e670 P 9081292597be4a72a42b216fddcd7971 reported cstate change: config changed from index 2 to 3, VOTER 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5) evicted. New cstate: current_term: 1 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:25.912077 27012 tablet_service.cc:1514] Processing DeleteTablet for tablet d6227a1f28614fbcb1b5a573d125e670 with delete_type TABLET_DATA_TOMBSTONED (TS 3b5968cebd0b4294898fdd89b7a5b3a1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46924
I20240501 19:57:25.912390 7918 tablet_replica.cc:330] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: stopping tablet replica
I20240501 19:57:25.912614 7918 raft_consensus.cc:2229] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:25.912806 7918 raft_consensus.cc:2258] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:25.913307 7918 ts_tablet_manager.cc:1875] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:25.915799 7918 ts_tablet_manager.cc:1888] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:25.915880 7918 log.cc:1196] T d6227a1f28614fbcb1b5a573d125e670 P 3b5968cebd0b4294898fdd89b7a5b3a1: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/wals/d6227a1f28614fbcb1b5a573d125e670
I20240501 19:57:25.916400 24438 catalog_manager.cc:4830] TS 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247): tablet d6227a1f28614fbcb1b5a573d125e670 (table rebalancer_extra_table_18 [id=5d329d80c3ab4a1e8f63df8d48a23642]) successfully deleted
I20240501 19:57:25.926458 6889 raft_consensus.cc:482] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:25.926584 6889 raft_consensus.cc:504] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:25.927067 6889 leader_election.cc:288] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:25.927436 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d80b9d644fb4160950462cdcc895052" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
W20240501 19:57:25.927879 24531 leader_election.cc:334] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:25.927948 24531 leader_election.cc:302] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:25.928172 6889 raft_consensus.cc:2726] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:25.952852 7836 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:25.952991 7836 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:25.953349 7836 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:25.956739 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:25.956858 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:25.956908 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:25.957618 7836 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:25.967967 5478 raft_consensus.cc:482] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:25.968094 5478 raft_consensus.cc:504] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:25.968503 5478 leader_election.cc:288] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:25.969148 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44d1713369478191bfc30457bf61c1" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
W20240501 19:57:25.970191 24924 leader_election.cc:334] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:25.970266 24924 leader_election.cc:302] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:57:25.970433 5478 raft_consensus.cc:2726] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:25.973742 7259 raft_consensus.cc:482] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:25.973852 7259 raft_consensus.cc:504] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:25.974237 7259 leader_election.cc:288] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3:38893)
I20240501 19:57:25.975580 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9d64cdfaeb2d46b29845160e395fc241" is_pre_election: true
W20240501 19:57:25.976429 27229 leader_election.cc:334] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:25.976505 27229 leader_election.cc:302] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 4a6e54e86a9949e5970b42e8e32e7561, 9d64cdfaeb2d46b29845160e395fc241
I20240501 19:57:25.976883 7184 raft_consensus.cc:2726] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:57:25.986644 5478 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:25.986758 5478 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:25.987196 5478 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:25.988853 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:25.988957 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:25.988999 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:25.989400 5478 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:57:25.990501 7259 raft_consensus.cc:482] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:25.990594 7259 raft_consensus.cc:504] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:25.990975 7259 leader_election.cc:288] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
W20240501 19:57:25.991932 27229 leader_election.cc:334] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:25.992027 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe96677d2140d1aa22870197a9890f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:25.992375 27227 leader_election.cc:302] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 4a6e54e86a9949e5970b42e8e32e7561, 9081292597be4a72a42b216fddcd7971
I20240501 19:57:25.992570 7259 raft_consensus.cc:2726] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.002904 6889 raft_consensus.cc:482] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.003002 6889 raft_consensus.cc:504] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.003418 6889 leader_election.cc:288] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:26.004032 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d72f3b14c471cbd6fcebe50afa8dd" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
W20240501 19:57:26.005223 24531 leader_election.cc:334] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.005298 24531 leader_election.cc:302] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:26.005525 6889 raft_consensus.cc:2726] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.045336 7427 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.045465 7427 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.045863 7427 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:26.047329 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:26.047592 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.047677 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.048266 24836 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:26.049219 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:26.049381 7427 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.050690 7259 raft_consensus.cc:482] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.050782 7259 raft_consensus.cc:504] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:26.052235 7259 leader_election.cc:288] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:26.053004 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:26.055497 7906 raft_consensus.cc:482] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.055610 7906 raft_consensus.cc:504] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.056025 7906 leader_election.cc:288] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:26.056408 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca5963c18944efe8e5a87cd06d970b6" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
W20240501 19:57:26.057097 27229 leader_election.cc:334] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.057176 27229 leader_election.cc:302] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 4a6e54e86a9949e5970b42e8e32e7561, b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:26.057380 7259 raft_consensus.cc:2726] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:26.057595 24662 leader_election.cc:334] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.057663 24662 leader_election.cc:302] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.057750 24707 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:57:26.058018 7906 raft_consensus.cc:2726] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.058674 27012 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:26.060187 24966 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 19:57:26.063985 24575 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:26.067631 7783 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.067739 7783 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.068094 7783 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:26.075316 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:26.075470 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.075524 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.076104 7783 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.076812 7427 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.076894 7427 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.077745 7427 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:26.079200 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.079217 7427 raft_consensus.cc:482] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
W20240501 19:57:26.079293 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.079325 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.079309 7427 raft_consensus.cc:504] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.079600 7427 leader_election.cc:288] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:26.079674 7427 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:26.080317 26967 leader_election.cc:334] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.086148 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca5963c18944efe8e5a87cd06d970b6" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
I20240501 19:57:26.086282 24725 raft_consensus.cc:2454] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:26.086625 26965 leader_election.cc:302] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.086824 7276 raft_consensus.cc:2781] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.086875 7276 raft_consensus.cc:482] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.086915 7276 raft_consensus.cc:3037] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.088100 7276 raft_consensus.cc:504] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.088410 7276 leader_election.cc:288] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299)
I20240501 19:57:26.090445 7906 raft_consensus.cc:482] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.090536 7906 raft_consensus.cc:504] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:26.090855 7906 leader_election.cc:288] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:26.091841 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe96677d2140d1aa22870197a9890f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:26.091962 27278 raft_consensus.cc:2454] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
W20240501 19:57:26.092267 24662 leader_election.cc:334] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.092429 24662 leader_election.cc:302] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.092746 7906 raft_consensus.cc:2781] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.092800 7906 raft_consensus.cc:482] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.092851 7906 raft_consensus.cc:3037] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.093711 7906 raft_consensus.cc:504] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:26.094013 7906 leader_election.cc:288] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:26.094197 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca5963c18944efe8e5a87cd06d970b6" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971"
I20240501 19:57:26.094290 24725 raft_consensus.cc:3037] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:26.095120 24662 leader_election.cc:334] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.095268 24725 raft_consensus.cc:2454] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 2.
I20240501 19:57:26.095587 26965 leader_election.cc:302] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:26.095598 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe96677d2140d1aa22870197a9890f" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:26.095702 27278 raft_consensus.cc:3037] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.095738 7276 raft_consensus.cc:2781] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.095820 7276 raft_consensus.cc:686] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:26.095954 7276 consensus_queue.cc:237] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.096689 27278 raft_consensus.cc:2454] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 2.
W20240501 19:57:26.097923 26967 leader_election.cc:334] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.099587 24662 leader_election.cc:302] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.099515 24453 catalog_manager.cc:5484] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 1 to 2, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 2 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:26.100234 7370 raft_consensus.cc:2781] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.100311 7370 raft_consensus.cc:686] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [term 2 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:26.100425 7370 consensus_queue.cc:237] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } }
I20240501 19:57:26.102790 24446 catalog_manager.cc:5484] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 1 to 2, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 2 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } }
I20240501 19:57:26.138361 7370 raft_consensus.cc:482] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.138478 7370 raft_consensus.cc:504] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.138849 7370 leader_election.cc:288] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:26.139261 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0c2b7abe74eb8b8bf30d8b3cf756d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:26.139389 27278 raft_consensus.cc:2454] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:26.139901 24662 leader_election.cc:302] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
W20240501 19:57:26.140031 24662 leader_election.cc:334] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.140117 7370 raft_consensus.cc:2781] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.140183 7370 raft_consensus.cc:482] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.140233 7370 raft_consensus.cc:3037] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.141167 7370 raft_consensus.cc:504] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.141538 7370 leader_election.cc:288] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:26.142159 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0c2b7abe74eb8b8bf30d8b3cf756d" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:26.142266 27278 raft_consensus.cc:3037] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:26.142478 24662 leader_election.cc:334] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.143095 27278 raft_consensus.cc:2454] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 2.
I20240501 19:57:26.143395 24662 leader_election.cc:302] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:26.144407 7906 raft_consensus.cc:2781] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.144488 7906 raft_consensus.cc:686] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [term 2 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:26.144600 7906 consensus_queue.cc:237] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.147097 24453 catalog_manager.cc:5484] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 1 to 2, leader changed from 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 2 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
I20240501 19:57:26.149633 7427 raft_consensus.cc:482] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.149724 7427 raft_consensus.cc:504] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:26.150030 7427 leader_election.cc:288] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:26.150455 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce9f9c2a5be47d3986e3d7b53570b12" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
W20240501 19:57:26.150951 26967 leader_election.cc:334] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.151141 26965 leader_election.cc:302] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:26.151305 7427 raft_consensus.cc:2726] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.192632 7783 raft_consensus.cc:482] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.192764 7783 raft_consensus.cc:504] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.193205 7783 leader_election.cc:288] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:26.193490 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d72f3b14c471cbd6fcebe50afa8dd" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:26.193608 24594 raft_consensus.cc:2454] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:26.193928 24922 leader_election.cc:302] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:26.194097 7783 raft_consensus.cc:2781] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.194149 7783 raft_consensus.cc:482] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.194190 7783 raft_consensus.cc:3037] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.195156 6889 raft_consensus.cc:482] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.195247 6889 raft_consensus.cc:504] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:26.195477 7783 raft_consensus.cc:504] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.195611 6889 leader_election.cc:288] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:26.195832 7783 leader_election.cc:288] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:26.196223 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce9f9c2a5be47d3986e3d7b53570b12" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" is_pre_election: true
W20240501 19:57:26.196303 24924 leader_election.cc:334] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.196359 27032 raft_consensus.cc:2454] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:26.196564 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453d72f3b14c471cbd6fcebe50afa8dd" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:26.196672 24529 leader_election.cc:302] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:26.196687 24594 raft_consensus.cc:3037] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.197317 7684 raft_consensus.cc:2781] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.197367 7684 raft_consensus.cc:482] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.197402 7684 raft_consensus.cc:3037] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.197775 24594 raft_consensus.cc:2454] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 2.
I20240501 19:57:26.198539 7684 raft_consensus.cc:504] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
W20240501 19:57:26.198822 24924 leader_election.cc:334] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.198894 7684 leader_election.cc:288] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:26.199553 24922 leader_election.cc:302] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:57:26.199729 7783 raft_consensus.cc:2781] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.199813 7783 raft_consensus.cc:686] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:26.199928 7783 consensus_queue.cc:237] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:26.200546 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce9f9c2a5be47d3986e3d7b53570b12" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1"
I20240501 19:57:26.200659 27032 raft_consensus.cc:3037] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.201726 27032 raft_consensus.cc:2454] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 2.
I20240501 19:57:26.201990 24529 leader_election.cc:302] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
I20240501 19:57:26.202471 7684 raft_consensus.cc:2781] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.202564 7684 raft_consensus.cc:686] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:26.202703 7684 consensus_queue.cc:237] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:26.203557 24446 catalog_manager.cc:5484] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 1 to 2, leader changed from 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 2 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } }
W20240501 19:57:26.206521 24531 leader_election.cc:334] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:26.206687 24531 leader_election.cc:334] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.208287 24453 catalog_manager.cc:5484] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 1 to 2, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 2 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:26.213871 8018 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.213958 8018 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:26.214294 8018 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:26.215917 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:26.216029 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.216082 24531 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.216478 8018 raft_consensus.cc:2726] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.223107 7427 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.223201 7427 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.223547 7427 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:26.224175 7427 raft_consensus.cc:482] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.224258 7427 raft_consensus.cc:504] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:26.224709 7427 leader_election.cc:288] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:26.225059 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44d1713369478191bfc30457bf61c1" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
W20240501 19:57:26.225109 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.225198 24987 raft_consensus.cc:2454] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
W20240501 19:57:26.225241 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.225283 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.225610 7427 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.225718 26966 leader_election.cc:302] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
W20240501 19:57:26.226018 26967 leader_election.cc:334] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.226498 7427 raft_consensus.cc:2781] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.226583 7427 raft_consensus.cc:482] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.226631 7427 raft_consensus.cc:3037] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.227821 7427 raft_consensus.cc:504] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:26.228109 7427 leader_election.cc:288] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: Requested vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:26.228443 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44d1713369478191bfc30457bf61c1" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:26.228562 24987 raft_consensus.cc:3037] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.229669 24987 raft_consensus.cc:2454] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 2.
I20240501 19:57:26.229950 26966 leader_election.cc:302] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:26.230983 7427 raft_consensus.cc:2781] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.231786 7427 raft_consensus.cc:686] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:26.232192 7427 consensus_queue.cc:237] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:26.234081 7427 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.234153 7427 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:26.234428 7427 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.236557 24451 catalog_manager.cc:5484] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 1 to 2, leader changed from 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6) to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 2 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
W20240501 19:57:26.238463 26967 leader_election.cc:334] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.239392 7783 raft_consensus.cc:482] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.239488 7783 raft_consensus.cc:504] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:26.239873 7783 leader_election.cc:288] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:26.240949 24924 leader_election.cc:334] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.241325 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:26.241470 27280 raft_consensus.cc:2454] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 1.
I20240501 19:57:26.241879 24924 leader_election.cc:302] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.242043 7783 raft_consensus.cc:2781] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.242136 7783 raft_consensus.cc:482] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.242187 7783 raft_consensus.cc:3037] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.243376 7783 raft_consensus.cc:504] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:26.243757 7783 leader_election.cc:288] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.244222 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1d950141e14ca68da6d8b3e8254552" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:26.244324 27280 raft_consensus.cc:3037] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:26.244755 24924 leader_election.cc:334] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:26.245093 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.245497 27280 raft_consensus.cc:2454] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9fd7c49d12b48a7ae73ab09876a0b82 in term 2.
W20240501 19:57:26.246373 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.246449 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.246739 7427 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:57:26.246954 24924 leader_election.cc:302] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.247215 7783 raft_consensus.cc:2781] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.247314 7783 raft_consensus.cc:686] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 LEADER]: Becoming Leader. State: Replica: b9fd7c49d12b48a7ae73ab09876a0b82, State: Running, Role: LEADER
I20240501 19:57:26.247474 7783 consensus_queue.cc:237] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:26.248284 7836 raft_consensus.cc:482] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.248399 7836 raft_consensus.cc:504] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:26.248816 7836 leader_election.cc:288] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.249119 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:26.249223 27280 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 2.
I20240501 19:57:26.249467 24793 leader_election.cc:302] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
I20240501 19:57:26.249977 7836 raft_consensus.cc:2781] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 19:57:26.250039 7836 raft_consensus.cc:482] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.250083 7836 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 2 FOLLOWER]: Advancing to term 3
W20240501 19:57:26.251179 24793 leader_election.cc:334] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.251482 24454 catalog_manager.cc:5484] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 reported cstate change: term changed from 1 to 2, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4). New cstate: current_term: 2 leader_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } }
I20240501 19:57:26.251626 7836 raft_consensus.cc:504] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:26.251912 7836 leader_election.cc:288] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 3 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.252590 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c18dfded64901bb4645af3dbe2dbc" candidate_uuid: "9d64cdfaeb2d46b29845160e395fc241" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:26.252701 27280 raft_consensus.cc:3037] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:26.253721 27280 raft_consensus.cc:2454] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d64cdfaeb2d46b29845160e395fc241 in term 3.
I20240501 19:57:26.253934 24793 leader_election.cc:302] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9d64cdfaeb2d46b29845160e395fc241; no voters:
W20240501 19:57:26.254246 24793 leader_election.cc:334] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.254261 7836 raft_consensus.cc:2781] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:26.254563 7836 raft_consensus.cc:686] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [term 3 LEADER]: Becoming Leader. State: Replica: 9d64cdfaeb2d46b29845160e395fc241, State: Running, Role: LEADER
I20240501 19:57:26.254698 7836 consensus_queue.cc:237] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:26.261644 24453 catalog_manager.cc:5484] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 reported cstate change: term changed from 2 to 3, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to 9d64cdfaeb2d46b29845160e395fc241 (127.17.65.3). New cstate: current_term: 3 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:26.265947 7783 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.266057 7783 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:26.266402 7783 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:26.269601 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:26.269935 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:26.270138 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.270200 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.270316 7783 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.280685 7783 raft_consensus.cc:482] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.280797 7783 raft_consensus.cc:504] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:26.281153 7783 leader_election.cc:288] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399)
I20240501 19:57:26.281566 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed0e67f44543d09dbeda7100c7e0d6" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
W20240501 19:57:26.283661 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:26.284459 24924 leader_election.cc:334] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.284598 24924 leader_election.cc:302] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 4a6e54e86a9949e5970b42e8e32e7561, bbf7735f82ca4f65a5170ac7909faa47
I20240501 19:57:26.284752 7783 raft_consensus.cc:2726] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.286108 7836 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:26.286207 7836 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:26.286542 7836 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:26.287663 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:26.287781 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.287833 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.288532 7836 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.290513 8018 raft_consensus.cc:482] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.290609 8018 raft_consensus.cc:504] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:26.290977 8018 leader_election.cc:288] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
I20240501 19:57:26.291425 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed0e67f44543d09dbeda7100c7e0d6" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:26.291554 24987 raft_consensus.cc:2454] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 1.
I20240501 19:57:26.291920 24530 leader_election.cc:302] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters:
W20240501 19:57:26.292040 24531 leader_election.cc:334] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.293262 7684 raft_consensus.cc:2781] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.293332 7684 raft_consensus.cc:482] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.293378 7684 raft_consensus.cc:3037] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.294520 7684 raft_consensus.cc:504] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:26.294911 7684 leader_election.cc:288] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: Requested vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:57:26.296229 24531 leader_election.cc:334] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.298539 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed0e67f44543d09dbeda7100c7e0d6" candidate_uuid: "bbf7735f82ca4f65a5170ac7909faa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:26.298643 24987 raft_consensus.cc:3037] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.299770 24987 raft_consensus.cc:2454] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf7735f82ca4f65a5170ac7909faa47 in term 2.
I20240501 19:57:26.300899 24530 leader_election.cc:302] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82, bbf7735f82ca4f65a5170ac7909faa47; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.301060 7684 raft_consensus.cc:2781] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.301141 7684 raft_consensus.cc:686] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 LEADER]: Becoming Leader. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Running, Role: LEADER
I20240501 19:57:26.301280 7684 consensus_queue.cc:237] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:26.304036 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:26.304597 24707 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:57:26.305900 24453 catalog_manager.cc:5484] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: term changed from 1 to 2, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1). New cstate: current_term: 2 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } }
I20240501 19:57:26.306818 27012 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 19:57:26.314352 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:26.315215 27267 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 19:57:26.316419 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:26.329337 7259 raft_consensus.cc:482] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.329463 7259 raft_consensus.cc:504] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:26.329823 7259 leader_election.cc:288] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.330250 24725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76585af218d4ebea389db62aa99a3f7" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9081292597be4a72a42b216fddcd7971" is_pre_election: true
W20240501 19:57:26.332125 27229 leader_election.cc:334] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.332203 27229 leader_election.cc:302] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 4a6e54e86a9949e5970b42e8e32e7561, 9081292597be4a72a42b216fddcd7971
I20240501 19:57:26.332401 7259 raft_consensus.cc:2726] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.340983 7783 raft_consensus.cc:482] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.341091 7783 raft_consensus.cc:504] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:26.341437 7783 leader_election.cc:288] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797)
I20240501 19:57:26.343139 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
W20240501 19:57:26.343475 24924 leader_election.cc:334] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.343648 24924 leader_election.cc:302] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 4a6e54e86a9949e5970b42e8e32e7561, 53e34bb09d8e41129a5683079ff63610
I20240501 19:57:26.343851 7783 raft_consensus.cc:2726] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:57:26.399889 7427 raft_consensus.cc:482] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.400015 7427 raft_consensus.cc:504] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.400488 7427 leader_election.cc:288] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.400868 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d80b9d644fb4160950462cdcc895052" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47" is_pre_election: true
I20240501 19:57:26.401047 24594 raft_consensus.cc:2454] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 1.
I20240501 19:57:26.401492 26965 leader_election.cc:302] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters:
W20240501 19:57:26.401582 26967 leader_election.cc:334] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.401839 7276 raft_consensus.cc:2781] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.401896 7276 raft_consensus.cc:482] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.401944 7276 raft_consensus.cc:3037] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.402915 7276 raft_consensus.cc:504] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.403270 7276 leader_election.cc:288] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: Requested vote from peers bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.403724 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d80b9d644fb4160950462cdcc895052" candidate_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbf7735f82ca4f65a5170ac7909faa47"
I20240501 19:57:26.403837 24594 raft_consensus.cc:3037] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:26.404417 26967 leader_election.cc:334] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.404417 7184 raft_consensus.cc:482] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.404525 7184 raft_consensus.cc:504] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:26.404837 24594 raft_consensus.cc:2454] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5968cebd0b4294898fdd89b7a5b3a1 in term 2.
I20240501 19:57:26.404894 7184 leader_election.cc:288] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.405136 26965 leader_election.cc:302] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1, bbf7735f82ca4f65a5170ac7909faa47; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.405493 7276 raft_consensus.cc:2781] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.405596 7276 raft_consensus.cc:686] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b5968cebd0b4294898fdd89b7a5b3a1, State: Running, Role: LEADER
I20240501 19:57:26.405730 7276 consensus_queue.cc:237] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
W20240501 19:57:26.406713 27229 leader_election.cc:334] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.406910 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" is_pre_election: true
I20240501 19:57:26.407078 24987 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 2.
I20240501 19:57:26.407354 27228 leader_election.cc:302] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.407685 7184 raft_consensus.cc:2781] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 19:57:26.407742 7184 raft_consensus.cc:482] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.407784 7184 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:26.408905 7184 raft_consensus.cc:504] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:26.409231 7184 leader_election.cc:288] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 election: Requested vote from peers b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.410483 24453 catalog_manager.cc:5484] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 reported cstate change: term changed from 1 to 2, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5). New cstate: current_term: 2 leader_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:26.413470 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433347ce828b4d17a70308c421ab4f1f" candidate_uuid: "53e34bb09d8e41129a5683079ff63610" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82"
I20240501 19:57:26.413600 24987 raft_consensus.cc:3037] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:26.414419 24987 raft_consensus.cc:2454] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e34bb09d8e41129a5683079ff63610 in term 3.
I20240501 19:57:26.415014 27228 leader_election.cc:302] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, b9fd7c49d12b48a7ae73ab09876a0b82; no voters:
I20240501 19:57:26.415361 7259 raft_consensus.cc:2781] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:26.415445 7259 raft_consensus.cc:686] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [term 3 LEADER]: Becoming Leader. State: Replica: 53e34bb09d8e41129a5683079ff63610, State: Running, Role: LEADER
I20240501 19:57:26.415598 7259 consensus_queue.cc:237] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
W20240501 19:57:26.418354 27229 leader_election.cc:334] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.419726 24454 catalog_manager.cc:5484] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 reported cstate change: term changed from 2 to 3, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to 53e34bb09d8e41129a5683079ff63610 (127.17.65.7). New cstate: current_term: 3 leader_uuid: "53e34bb09d8e41129a5683079ff63610" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } }
I20240501 19:57:26.425882 7370 raft_consensus.cc:482] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.425992 7370 raft_consensus.cc:504] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:26.426385 7370 leader_election.cc:288] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.427363 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76585af218d4ebea389db62aa99a3f7" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610" is_pre_election: true
I20240501 19:57:26.427508 27280 raft_consensus.cc:2454] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 1.
I20240501 19:57:26.427932 24662 leader_election.cc:302] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters:
I20240501 19:57:26.428243 7370 raft_consensus.cc:2781] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:26.428300 7370 raft_consensus.cc:482] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.428349 7370 raft_consensus.cc:3037] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:26.429571 7370 raft_consensus.cc:504] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:26.429912 7370 leader_election.cc:288] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: Requested vote from peers 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:26.431140 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76585af218d4ebea389db62aa99a3f7" candidate_uuid: "9081292597be4a72a42b216fddcd7971" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "53e34bb09d8e41129a5683079ff63610"
I20240501 19:57:26.431257 27280 raft_consensus.cc:3037] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:26.431453 24662 leader_election.cc:334] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.432372 27280 raft_consensus.cc:2454] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9081292597be4a72a42b216fddcd7971 in term 2.
W20240501 19:57:26.432731 24662 leader_election.cc:334] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.432901 24662 leader_election.cc:302] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610, 9081292597be4a72a42b216fddcd7971; no voters: 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.433116 7906 raft_consensus.cc:2781] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:26.433198 7906 raft_consensus.cc:686] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [term 2 LEADER]: Becoming Leader. State: Replica: 9081292597be4a72a42b216fddcd7971, State: Running, Role: LEADER
I20240501 19:57:26.433326 7906 consensus_queue.cc:237] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:26.435866 24452 catalog_manager.cc:5484] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 reported cstate change: term changed from 1 to 2, leader changed from 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8) to 9081292597be4a72a42b216fddcd7971 (127.17.65.2). New cstate: current_term: 2 leader_uuid: "9081292597be4a72a42b216fddcd7971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } }
I20240501 19:57:26.448861 7259 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.448984 7259 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:26.449401 7259 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:26.451695 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:26.451810 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.451854 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.452093 7259 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.456252 7906 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.456362 7906 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:26.456949 7906 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:57:26.457697 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:57:26.458572 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:26.458649 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:57:26.458758 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.459167 7906 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.463714 7259 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:26.463817 7259 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:26.464205 7259 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:26.465504 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:26.465626 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.465669 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:26.465797 7259 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.517959 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:26.519903 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:26.521124 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:26.523209 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:26.524025 24725 raft_consensus.cc:1261] T dca5963c18944efe8e5a87cd06d970b6 P 9081292597be4a72a42b216fddcd7971 [term 2 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:57:26.524441 7427 consensus_queue.cc:1035] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:26.530411 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:26.536329 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 19:57:26.539736 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:26.608920 27280 raft_consensus.cc:1261] T bcbe96677d2140d1aa22870197a9890f P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:57:26.609428 7906 consensus_queue.cc:1035] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:26.610342 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:26.612869 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:26.636781 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:26.641567 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:26.655565 27280 raft_consensus.cc:1261] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:57:26.657881 6762 consensus_queue.cc:1035] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:26.660398 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:26.678833 24987 raft_consensus.cc:1261] T 2fed0e67f44543d09dbeda7100c7e0d6 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:26.679242 6889 consensus_queue.cc:1035] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:26.683811 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:26.697388 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:26.712109 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:26.748742 27031 raft_consensus.cc:1261] T fce9f9c2a5be47d3986e3d7b53570b12 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:26.751993 7684 consensus_queue.cc:1035] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:26.760967 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:26.770512 24987 raft_consensus.cc:1261] T 2a44d1713369478191bfc30457bf61c1 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:26.770923 7276 consensus_queue.cc:1035] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:26.773862 24594 raft_consensus.cc:1261] T 453d72f3b14c471cbd6fcebe50afa8dd P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:57:26.775894 27280 raft_consensus.cc:1261] T ca1d950141e14ca68da6d8b3e8254552 P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer b9fd7c49d12b48a7ae73ab09876a0b82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:26.776075 7783 consensus_queue.cc:1035] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:26.776654 7783 consensus_queue.cc:1035] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:26.780545 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:26.784744 27278 raft_consensus.cc:1261] T 4a4c18dfded64901bb4645af3dbe2dbc P 53e34bb09d8e41129a5683079ff63610 [term 3 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 19:57:26.785281 5887 consensus_queue.cc:1035] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:26.817494 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:26.845544 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:26.856088 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:26.859262 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:26.865729 24594 raft_consensus.cc:1261] T 5d80b9d644fb4160950462cdcc895052 P bbf7735f82ca4f65a5170ac7909faa47 [term 2 FOLLOWER]: Refusing update from remote peer 3b5968cebd0b4294898fdd89b7a5b3a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:26.866453 7427 consensus_queue.cc:1035] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:26.867134 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:26.879483 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:26.886849 27280 raft_consensus.cc:1261] T c76585af218d4ebea389db62aa99a3f7 P 53e34bb09d8e41129a5683079ff63610 [term 2 FOLLOWER]: Refusing update from remote peer 9081292597be4a72a42b216fddcd7971: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:57:26.887518 6762 consensus_queue.cc:1035] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:57:26.906410 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:26.908192 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:26.913520 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:26.966423 24858 consensus_queue.cc:237] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:26.968024 27031 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:26.968026 27280 raft_consensus.cc:1261] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Refusing update from remote peer 9d64cdfaeb2d46b29845160e395fc241: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:26.968397 5887 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:26.968571 7836 consensus_queue.cc:1035] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:26.970504 8190 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 9d64cdfaeb2d46b29845160e395fc241 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:26.970762 27280 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:26.970996 27031 raft_consensus.cc:2932] T 9acd3c8577694e799d3f67a0f2f6adcc P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:26.975790 24446 catalog_manager.cc:5484] T 9acd3c8577694e799d3f67a0f2f6adcc P 53e34bb09d8e41129a5683079ff63610 reported cstate change: config changed from index 6 to 7, VOTER 9081292597be4a72a42b216fddcd7971 (127.17.65.2) evicted. New cstate: current_term: 1 leader_uuid: "9d64cdfaeb2d46b29845160e395fc241" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } }
I20240501 19:57:26.980703 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 9acd3c8577694e799d3f67a0f2f6adcc with delete_type TABLET_DATA_TOMBSTONED (TS 9081292597be4a72a42b216fddcd7971 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59008
I20240501 19:57:26.981436 8201 tablet_replica.cc:330] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: stopping tablet replica
I20240501 19:57:26.981917 8201 raft_consensus.cc:2229] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:26.982218 8201 raft_consensus.cc:2258] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:26.982795 8201 ts_tablet_manager.cc:1875] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:26.987107 8201 ts_tablet_manager.cc:1888] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:26.987195 8201 log.cc:1196] T 9acd3c8577694e799d3f67a0f2f6adcc P 9081292597be4a72a42b216fddcd7971: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/wals/9acd3c8577694e799d3f67a0f2f6adcc
I20240501 19:57:26.987694 24438 catalog_manager.cc:4830] TS 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): tablet 9acd3c8577694e799d3f67a0f2f6adcc (table rebalance_test_table_19 [id=4ce9bced17984c458d3757f7e32f02a2]) successfully deleted
I20240501 19:57:27.000238 24987 raft_consensus.cc:1261] T 433347ce828b4d17a70308c421ab4f1f P b9fd7c49d12b48a7ae73ab09876a0b82 [term 3 FOLLOWER]: Refusing update from remote peer 53e34bb09d8e41129a5683079ff63610: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 19:57:27.000633 7184 consensus_queue.cc:1035] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 19:57:27.079919 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.089947 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.117985 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:27.119092 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:27.127260 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:27.130357 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:27.131213 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:27.136444 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 19:57:27.147574 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.173274 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:27.173610 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:27.209455 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.210927 24594 consensus_queue.cc:237] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:27.212112 24725 raft_consensus.cc:1261] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:27.212112 24858 raft_consensus.cc:1261] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Refusing update from remote peer bbf7735f82ca4f65a5170ac7909faa47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:27.212520 8018 consensus_queue.cc:1035] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:27.212911 8018 consensus_queue.cc:1035] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:27.214489 6889 raft_consensus.cc:2932] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:27.216825 24725 raft_consensus.cc:2932] T dd1df8a61b1c41368727514e30e263a8 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:27.218871 24858 raft_consensus.cc:2932] T dd1df8a61b1c41368727514e30e263a8 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
W20240501 19:57:27.221971 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.221208 24454 catalog_manager.cc:5484] T dd1df8a61b1c41368727514e30e263a8 P bbf7735f82ca4f65a5170ac7909faa47 reported cstate change: config changed from index 2 to 3, VOTER 53e34bb09d8e41129a5683079ff63610 (127.17.65.7) evicted. New cstate: current_term: 1 leader_uuid: "bbf7735f82ca4f65a5170ac7909faa47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } } }
I20240501 19:57:27.225600 27267 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1df8a61b1c41368727514e30e263a8 with delete_type TABLET_DATA_TOMBSTONED (TS 53e34bb09d8e41129a5683079ff63610 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49040
I20240501 19:57:27.225989 8268 tablet_replica.cc:330] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: stopping tablet replica
I20240501 19:57:27.226316 8268 raft_consensus.cc:2229] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:27.227167 8268 raft_consensus.cc:2258] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:27.227653 8268 ts_tablet_manager.cc:1875] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:27.230132 8268 ts_tablet_manager.cc:1888] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 19:57:27.230216 8268 log.cc:1196] T dd1df8a61b1c41368727514e30e263a8 P 53e34bb09d8e41129a5683079ff63610: Deleting WAL directory at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/wals/dd1df8a61b1c41368727514e30e263a8
I20240501 19:57:27.231081 24440 catalog_manager.cc:4830] TS 53e34bb09d8e41129a5683079ff63610 (127.17.65.7:39797): tablet dd1df8a61b1c41368727514e30e263a8 (table rebalancer_extra_table_38 [id=c08a0985c3ef425fbe36648030168464]) successfully deleted
W20240501 19:57:27.246441 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.252529 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.309796 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.319065 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.335022 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.343641 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.356952 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.357282 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 19:57:27.357581 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.365012 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:27.366910 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:27.367560 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:27.377635 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 19:57:27.390334 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.392952 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.395741 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 19:57:27.400861 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.428732 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.466794 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.468262 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.475009 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.475219 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.490396 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.493011 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.502609 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.523519 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.536657 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.536911 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.563661 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.564291 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:27.564452 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 19:57:27.565851 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.566077 8154 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:27.566108 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:27.566169 8154 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:27.566606 8154 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:27.568717 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:27.568830 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:27.568871 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:27.569146 8155 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:27.572082 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.576318 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:27.576833 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:27.577021 5887 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:27.577117 5887 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:27.577423 5887 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:27.578596 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.579355 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 19:57:27.580749 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.581125 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:27.581432 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:27.581492 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:27.581730 5887 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:27.587944 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.589941 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.600694 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.600802 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.609038 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:27.613729 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.622920 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.628515 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.635370 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.666676 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.667068 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.669802 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.674722 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.679770 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.680178 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.684710 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.698917 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.699316 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.706372 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.707444 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.717005 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:27.722563 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.729188 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.731719 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.733397 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.748433 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.752270 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.761317 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.765141 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.781301 7427 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:27.781421 7427 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:27.781816 7427 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:27.784446 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:27.784562 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:27.784603 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:27.785389 7427 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:27.791870 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.798580 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:27.799144 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:27.801116 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:27.813943 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:27.815193 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:27.816356 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 19:57:27.824976 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.834801 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.863799 8155 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:27.863924 8155 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:27.864292 8155 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:27.865348 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:27.865576 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:27.865643 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:27.865785 8155 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:27.867398 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.881284 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.906886 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.909780 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.912137 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:27.914611 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:27.943218 8155 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:27.943357 8155 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:27.943758 8155 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:27.945871 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:27.945989 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:27.946040 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:27.946221 8155 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:27.976982 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:27.978526 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:27.988655 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:28.008013 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:28.008186 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:28.013335 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:28.030938 7427 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:28.031140 7427 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:28.031472 7427 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:28.032438 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:28.032544 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:28.032579 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:28.032783 7427 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:57:28.040380 7427 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:28.040479 7427 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:28.040815 7427 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:28.042495 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:28.042618 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:28.042670 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:28.042876 7427 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:28.043397 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:28.049065 7906 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:28.049193 7906 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:28.049536 7906 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:28.050704 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:28.050935 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:28.050995 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:28.051177 7906 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:28.065694 8190 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:28.065820 8190 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:28.066195 8190 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:28.067258 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:28.067363 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:28.067401 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:28.067528 8190 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:28.106231 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:28.148658 7684 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:28.148789 7684 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:28.149163 7684 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:28.150702 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:28.150825 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:28.150885 24531 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:28.151279 7684 raft_consensus.cc:2726] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:28.156509 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:28.177366 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:28.177655 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:28.179692 7184 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:28.179821 7184 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:28.180244 7184 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
I20240501 19:57:28.182160 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 19:57:28.184245 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:28.184370 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:28.184432 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:28.186197 7184 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:28.193569 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:28.203425 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 19:57:28.204876 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:28.207598 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:28.207803 8370 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:28.207887 8370 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:28.208252 8370 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:57:28.209064 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240501 19:57:28.211068 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:28.211174 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:28.211208 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:28.211454 8370 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:28.221757 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 19:57:28.240918 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:28.337405 7184 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:28.337522 7184 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:28.337903 7184 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:28.340816 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:28.341099 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:28.341166 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:28.341310 7184 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:28.376029 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:28.378626 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:28.384433 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:28.391211 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:28.392999 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:28.414255 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:28.573846 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 19:57:28.579012 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:57:28.581929 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:28.582311 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:28.584246 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:57:28.602679 7906 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:28.602820 7906 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:28.603281 7906 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:57:28.603865 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:57:28.605979 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:28.606060 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:57:28.606196 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:28.606388 7906 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:28.612303 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:57:28.678319 27031 consensus_queue.cc:237] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
W20240501 19:57:28.679942 26965 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer bbf7735f82ca4f65a5170ac7909faa47 (127.17.65.1:43399): Couldn't send request to peer bbf7735f82ca4f65a5170ac7909faa47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97f518e4e32644dc8a4af48941f85f55. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:28.680076 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:28.680186 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:28.687114 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:28.713469 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:28.922775 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.053570 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.085148 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.137646 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.138479 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.143580 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:29.150329 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:29.155241 8683 ts_tablet_manager.cc:902] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: Initiating tablet copy from peer 3b5968cebd0b4294898fdd89b7a5b3a1 (127.17.65.5:45247)
I20240501 19:57:29.155689 8683 tablet_copy_client.cc:308] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.65.5:45247
I20240501 19:57:29.156168 27052 tablet_copy_service.cc:140] P 3b5968cebd0b4294898fdd89b7a5b3a1: Received BeginTabletCopySession request for tablet 97f518e4e32644dc8a4af48941f85f55 from peer bbf7735f82ca4f65a5170ac7909faa47 ({username='slave'} at 127.17.65.1:60025)
I20240501 19:57:29.156265 27052 tablet_copy_service.cc:161] P 3b5968cebd0b4294898fdd89b7a5b3a1: Beginning new tablet copy session on tablet 97f518e4e32644dc8a4af48941f85f55 from peer bbf7735f82ca4f65a5170ac7909faa47 at {username='slave'} at 127.17.65.1:60025: session id = bbf7735f82ca4f65a5170ac7909faa47-97f518e4e32644dc8a4af48941f85f55
I20240501 19:57:29.157030 27052 tablet_copy_source_session.cc:205] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:29.157936 8683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f518e4e32644dc8a4af48941f85f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.159967 8683 tablet_copy_client.cc:783] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:29.160123 8683 tablet_copy_client.cc:647] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:29.161110 8683 tablet_copy_client.cc:523] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:29.162391 8683 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap starting.
W20240501 19:57:29.165648 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:29.166229 8683 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:57:29.166596 8683 tablet_bootstrap.cc:492] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: Bootstrap complete.
I20240501 19:57:29.166734 8683 ts_tablet_manager.cc:1367] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:29.167214 8683 raft_consensus.cc:348] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:29.167459 8683 raft_consensus.cc:729] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7735f82ca4f65a5170ac7909faa47, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.167562 8683 consensus_queue.cc:260] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:29.167820 8683 ts_tablet_manager.cc:1398] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:29.168304 27052 tablet_copy_service.cc:342] P 3b5968cebd0b4294898fdd89b7a5b3a1: Request end of tablet copy session bbf7735f82ca4f65a5170ac7909faa47-97f518e4e32644dc8a4af48941f85f55 received from {username='slave'} at 127.17.65.1:60025
I20240501 19:57:29.168411 27052 tablet_copy_service.cc:433] P 3b5968cebd0b4294898fdd89b7a5b3a1: ending tablet copy session bbf7735f82ca4f65a5170ac7909faa47-97f518e4e32644dc8a4af48941f85f55 on tablet 97f518e4e32644dc8a4af48941f85f55 with peer bbf7735f82ca4f65a5170ac7909faa47
W20240501 19:57:29.225541 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.265657 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.273486 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:29.342777 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.371958 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.482555 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:29.508525 8689 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:29.508639 8689 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:29.509104 8689 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:29.510037 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:29.510164 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:29.510216 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:29.510344 8689 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:29.577812 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:29.612766 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.624112 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:29.624527 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.690543 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:29.693831 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.718988 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:29.735786 24594 raft_consensus.cc:1203] T 97f518e4e32644dc8a4af48941f85f55 P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:57:29.738131 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.742760 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.744411 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:29.760811 5887 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:29.760911 5887 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:29.761265 5887 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:29.762291 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:29.762387 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:29.762441 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:29.762567 5887 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:29.769506 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.777221 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.781375 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.788107 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.804199 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.810827 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.814596 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.825950 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.835548 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.887910 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.905570 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.910277 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.921028 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.960304 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.962225 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.962253 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.969866 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.977746 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.981931 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:29.997215 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.009146 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.016242 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.031255 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.032637 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.042028 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.042321 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.048713 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.066781 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.093364 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.094336 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.116737 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:30.147614 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.149022 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.173470 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:30.210583 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.223482 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.223764 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.225219 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.241147 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.241272 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.247108 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.263015 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 19:57:30.263109 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240501 19:57:30.263932 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.273420 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 218 similar messages]
W20240501 19:57:30.274012 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.281045 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:30.281304 8690 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:30.281404 8690 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:30.281772 8690 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:30.282161 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240501 19:57:30.282744 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:30.282860 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:30.282896 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:30.283001 8690 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:30.290863 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 188 similar messages]
W20240501 19:57:30.299122 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240501 19:57:30.315176 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.317504 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.369700 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:30.370301 8689 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:30.370393 8689 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:30.370702 8689 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:30.371677 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:30.371783 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:30.371826 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:30.371949 8689 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:30.387280 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.397890 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.410463 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.417099 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:30.443054 8681 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:30.443190 8681 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:30.443818 8681 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:30.444851 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:30.444976 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:30.445024 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:30.445145 8681 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:30.445230 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.483876 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.493808 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:30.513074 8681 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:30.513192 8681 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:30.513623 8681 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:30.514492 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:30.514607 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:30.514657 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:30.514776 8681 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:30.526091 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.532927 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:30.541620 5887 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:30.541756 5887 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:30.542166 5887 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:30.543095 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:30.543205 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:30.543253 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:30.543366 5887 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:30.551823 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.563010 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.585427 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:30.597999 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.606417 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.628232 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.651991 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:30.714215 8682 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:30.714341 8682 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:30.714774 8682 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:30.715704 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:30.715854 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:30.715911 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:30.716060 8690 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:30.759600 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.766574 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:30.818331 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:30.897830 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:30.921957 7427 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:30.922063 7427 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:30.922375 7427 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:30.923363 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:30.923492 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:30.923542 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:30.923681 7427 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:30.963779 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:57:31.050629 7427 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:31.050743 7427 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:31.051173 7427 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:31.052058 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:31.052157 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:31.052206 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:31.052341 7427 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:31.066279 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:31.096504 6889 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:31.096604 6889 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:31.096922 6889 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:31.097779 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:31.097891 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:31.097951 24531 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:31.098086 6889 raft_consensus.cc:2726] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:31.144822 8694 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:31.144935 8694 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:31.145296 8694 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:31.146359 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:31.146678 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:31.146791 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:31.146947 8694 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:31.160025 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:31.269493 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:31.292954 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:31.335497 7906 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:31.335608 7906 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:31.335945 7906 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:31.336807 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:31.336900 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:31.336962 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:31.337105 7906 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:31.455834 7906 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:31.455963 7906 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:31.456421 7906 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:57:31.457477 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:31.457594 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
I20240501 19:57:31.457657 24661 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:31.457782 7906 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:31.457824 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:31.599267 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.618104 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:31.641561 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.669199 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.678632 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.693774 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.694779 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.713757 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.754323 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.777827 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:31.831940 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.911548 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:31.968431 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.013502 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:32.014252 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.075312 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.094666 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.098501 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.126327 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.138564 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.241243 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:32.282209 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.290555 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.299726 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.355232 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.358219 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.379989 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.380565 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.397922 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:32.407739 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.409933 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.415004 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.416033 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.423813 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.461745 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.465534 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.483390 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.484377 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.484643 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.486148 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.498354 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.503994 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.504618 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.515834 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.520570 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.524906 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.561709 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.567641 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.575276 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.614859 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.616895 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.619151 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.639218 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:32.684640 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.685735 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.698069 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.700911 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.709702 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.716171 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.723816 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.732549 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.747151 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.761936 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.772220 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.775276 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.787381 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.798481 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.799634 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.808414 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.818563 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:32.826750 8710 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:32.826853 8710 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:32.827307 8710 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:32.828518 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:32.828634 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:32.828684 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:32.828830 8710 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:32.916337 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.938278 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:32.939908 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.958171 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:32.979938 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.011119 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.038355 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.042448 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.069622 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.078510 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:33.082875 8717 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:33.082968 8717 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:33.083408 8717 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:33.084379 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:33.084506 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:33.084551 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:33.084663 8717 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:33.097695 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.123147 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:33.166023 8715 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:33.166160 8715 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:33.166551 8715 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:33.167619 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:33.167742 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:33.167788 24531 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:33.167917 8715 raft_consensus.cc:2726] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:33.183645 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.212173 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:33.212810 8717 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:33.212899 8717 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:33.213348 8717 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:57:33.213951 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:57:33.214229 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:33.214284 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:33.214408 8717 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:33.214411 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:33.215227 8711 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:33.215317 8711 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:33.215682 8711 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:33.216854 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:33.217015 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:33.217088 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:33.217213 8711 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:33.259008 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:33.265646 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.349282 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.363375 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:33.405898 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:33.465831 8713 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:33.465950 8713 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:33.466392 8713 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:33.467299 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:33.467406 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:33.467456 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:33.467590 8713 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:33.540230 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.540359 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:33.558992 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:33.621672 8721 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:33.621778 8721 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:33.622108 8721 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:33.623231 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:33.623378 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:33.623435 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:33.623587 8721 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:33.750370 8721 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:33.750490 8721 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:33.750818 8721 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:33.751965 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:33.752117 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:33.752178 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:33.752316 8721 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:33.790912 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:57:33.793644 8720 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:33.793766 8720 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:33.794158 8720 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:33.796411 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:33.796540 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:33.796595 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:33.796725 8720 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:33.799239 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:33.897300 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:33.976197 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:34.009409 8694 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:34.009547 8694 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:34.009963 8694 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:34.010871 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:34.010957 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:34.011003 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:34.011140 8694 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:34.037071 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:34.051651 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:34.138793 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.157972 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:34.190742 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:34.197647 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:34.267406 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:34.338469 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:34.409013 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:34.414242 8721 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:34.414351 8721 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:34.414697 8721 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:34.416693 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:34.416806 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:34.416859 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:34.417125 8710 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:34.433727 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.444505 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:34.444537 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:34.462148 8694 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:34.462291 8694 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:34.462740 8694 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:34.463886 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:34.464027 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:34.464092 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:34.464251 8694 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:34.516013 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:34.655278 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.659590 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.706840 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:34.728021 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.740691 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.754228 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.757365 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.780897 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.794726 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:34.802479 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.811328 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.826820 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.843250 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.886770 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:57:34.894757 8690 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:34.894869 8690 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:34.895283 8690 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:34.896243 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:34.896399 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:34.896436 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:57:34.896454 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:34.896585 8690 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:34.911900 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.933905 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.937721 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.946287 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.963883 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.970115 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.972911 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.978981 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.984412 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:34.992568 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.000561 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.009220 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.012250 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.012750 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.019914 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:35.043498 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.044236 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.046943 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.094033 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.126264 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.136839 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.143577 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.163100 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:35.163605 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.167392 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.167857 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.184795 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.197139 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.217896 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.232558 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.247889 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.255878 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.266222 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.272784 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 19:57:35.277123 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 199 similar messages]
W20240501 19:57:35.297618 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240501 19:57:35.298563 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240501 19:57:35.299135 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:57:35.306567 8711 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:35.306674 8711 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:35.307071 8711 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:35.308084 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:35.308187 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:35.308264 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:35.308382 8711 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:35.316490 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240501 19:57:35.318867 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 177 similar messages]
W20240501 19:57:35.339146 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.349313 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.366559 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.380991 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.386929 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.396369 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.444458 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.449045 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.451324 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.463853 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.475548 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.485589 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.487820 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.488566 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.496987 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.512375 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.575949 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.613344 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:35.698042 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:35.724083 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.796329 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.840157 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.846000 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:35.883615 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:57:35.938477 8717 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:35.938587 8717 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:35.939086 8717 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:57:35.939667 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:57:35.939949 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:35.940034 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:35.940162 8717 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:35.940168 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:35.964051 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.000552 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:36.049082 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:57:36.078928 8729 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:36.079058 8729 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:36.079438 8729 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:36.080446 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:36.080789 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:36.080842 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:36.080972 8729 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:36.100740 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:36.108441 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.220902 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:36.246637 8729 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:36.246783 8729 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:36.247184 8729 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:36.248929 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:36.249032 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:36.249074 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:36.249214 8729 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:36.379855 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:36.439059 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.454753 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:57:36.475378 8717 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:36.475522 8717 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:36.475957 8717 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:36.477025 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:36.477133 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:36.477180 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:36.477376 8717 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:36.519788 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.528123 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:36.574517 8694 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:36.574760 8694 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:36.575217 8694 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:36.576303 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:36.576447 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:36.576503 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:36.576709 8694 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:36.582499 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.600176 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.617580 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:36.702416 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.717896 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.746479 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:36.776324 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:36.826990 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:37.012459 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:37.029103 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:37.059226 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.059319 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.081459 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:37.124809 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.183245 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:37.209120 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.232434 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.262331 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:57:37.265898 8721 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:37.266072 8721 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:37.266434 8721 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:37.267632 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:37.267760 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:37.267807 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:37.267920 8721 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:37.301481 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.303298 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.309747 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.329875 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.333218 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.343566 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.354627 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.358320 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.366925 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.378193 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.410449 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:37.436827 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.438122 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:37.450388 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.451757 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.455765 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.465042 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.469291 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.473369 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.491845 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.511917 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.519625 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.530570 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.555976 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.573102 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.577487 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.581667 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:37.581722 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.583901 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.637916 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.674836 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.709075 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.719691 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.725132 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.726197 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.730072 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.732369 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.736831 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.755020 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.770349 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.778585 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.797119 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.804114 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.813041 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.867458 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.880920 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.889567 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.896467 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.908965 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.928207 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.938418 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.952422 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.964534 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.978633 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:37.986963 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:37.997635 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.007901 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.016336 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:38.028496 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.053099 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.068490 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.086793 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.119020 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.261445 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.297369 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:57:38.342149 8740 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:38.342504 8740 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:38.342870 8740 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:38.343717 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:38.343827 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:38.343868 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:38.343973 8740 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:38.406546 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:38.441557 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:38.454249 8690 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:38.454381 8690 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:38.454772 8690 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:38.455747 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:38.455857 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:38.455909 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:38.456120 8690 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:38.558337 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.586761 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.618705 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.655843 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:38.718920 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.891734 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:38.900213 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:38.903611 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:38.927640 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:38.928647 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:39.000567 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:39.080089 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:57:39.149281 8747 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:39.149400 8747 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:39.149907 8747 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:39.151618 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:39.151767 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:39.151822 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:39.151949 8747 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:39.172264 8737 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:39.172381 8737 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:39.172768 8737 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:39.173749 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:39.173867 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:39.173923 24531 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:39.174054 8737 raft_consensus.cc:2726] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:39.176318 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.225651 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:39.229703 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.280396 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:39.290774 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:39.300138 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:39.391603 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:39.418443 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:39.548159 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:39.550987 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.595168 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:39.628427 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.632699 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.656180 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.678519 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:39.679325 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.731513 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.818784 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.820638 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.826089 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.841745 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.845906 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.846768 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.849040 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.903503 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.907905 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.908727 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.916538 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.930480 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.930985 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:39.950465 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.950485 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.965682 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.984643 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:39.986992 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.002729 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.006700 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.007761 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.008010 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:57:40.009446 8751 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:40.009550 8751 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:40.009925 8751 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:40.011332 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:40.011472 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:40.011524 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:40.011691 8751 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:40.052578 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.064054 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.089278 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.100339 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.112295 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.119549 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.131289 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.138474 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:40.164749 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:57:40.172114 8690 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:40.172235 8690 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:40.172648 8690 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:40.173981 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:40.174100 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:40.174163 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:40.174368 8690 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:40.175163 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.181447 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.189246 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.203008 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.219107 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.232283 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.300446 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 19:57:40.301729 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 202 similar messages]
W20240501 19:57:40.302253 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.305886 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 151 similar messages]
I20240501 19:57:40.312623 8751 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:40.312705 8751 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:40.313040 8751 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:40.313935 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:40.314075 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:40.314136 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:40.314258 8751 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:40.315202 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.316870 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.323532 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 19:57:40.323809 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240501 19:57:40.324337 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.330675 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.337447 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240501 19:57:40.339422 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:40.348471 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.351320 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.352753 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.353891 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.365667 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.410990 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.461668 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:57:40.474869 8740 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:40.474979 8740 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:40.475359 8740 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:40.476414 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:40.476569 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:40.476644 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:40.476796 8740 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:40.478835 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.479475 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.483356 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.514364 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.517107 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.523145 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.546213 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.550035 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.560688 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.608261 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.611353 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.742379 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:40.808164 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:40.855895 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:57:40.891144 8690 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:40.891258 8690 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:40.891602 8690 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:40.894001 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:40.894219 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:40.894289 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:40.894589 8690 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:40.906237 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:40.946244 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.140419 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.140895 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:41.147217 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.243058 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.258555 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:41.301059 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:41.324457 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.429802 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.447110 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.457819 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.469113 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.496724 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.497932 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:41.608040 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:41.627337 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:41.731709 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:41.742591 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:41.755133 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:41.779819 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:41.821298 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:42.027734 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.064195 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.082091 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:42.123725 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:42.128226 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.136093 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.155615 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.157873 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.169070 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:42.184877 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.196879 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.228868 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.264820 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:42.266844 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.268301 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.296015 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.307933 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.309119 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:42.332829 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.346691 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.374644 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:57:42.381901 8755 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:42.382037 8755 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:42.382395 8755 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:42.383428 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:42.383551 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:42.383602 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:42.383726 8755 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:42.387240 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.412909 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.430979 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.436465 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.477171 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.477272 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.487562 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.501950 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.515360 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.525727 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.571055 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:42.580067 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.585743 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.586767 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.597959 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.609194 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.638994 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.660981 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.695729 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.706188 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.720304 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.728075 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.733794 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.764199 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:42.796157 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.799988 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.800233 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.817029 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.822387 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.822607 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.830662 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.841912 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.847178 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.847561 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.874390 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:42.874778 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.879807 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.904565 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.946301 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.954124 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:42.972018 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.026003 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.047573 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:57:43.059832 8729 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:43.059939 8729 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:43.060340 8729 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:43.061615 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:43.061735 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:43.061791 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:43.061914 8729 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:43.078109 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.081467 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:43.093704 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.124476 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.151000 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.155813 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.237949 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.260612 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.282752 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:43.475939 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:43.481799 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.488559 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.548516 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:43.638288 8758 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:43.638635 8758 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
W20240501 19:57:43.638729 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:57:43.639256 8758 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:43.640136 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:43.640249 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:43.640300 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:43.640519 8758 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:43.670207 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.748520 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.783324 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.805996 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:43.827113 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:43.918076 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:43.934763 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:44.014576 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:44.031147 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:44.108058 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:44.114503 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:44.138597 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:44.312621 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.342077 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.418149 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:44.428651 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:44.431962 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:44.434964 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.457020 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:44.476498 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.495827 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.496811 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:44.505637 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.530865 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.539436 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:44.605329 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.629209 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.632661 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.642014 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.668363 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.694247 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.711827 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.719530 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.727203 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.797232 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.813467 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.819774 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:44.855209 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.880620 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.887351 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.893666 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:57:44.957123 8763 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:44.957230 8763 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:44.957597 8763 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:44.958866 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:44.959491 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:44.959540 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:44.959652 8763 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:44.964236 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.964565 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.983803 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.983911 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.986387 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:44.997249 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.009297 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.011835 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:45.014395 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.017669 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.025631 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.100183 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.123189 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.130636 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.138752 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.142261 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.156109 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:57:45.223593 8774 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:45.223750 8774 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:45.224309 8774 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:57:45.225003 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:57:45.225664 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:45.225740 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:57:45.225862 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:45.225868 8774 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:45.253669 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.260151 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.261651 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.270458 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.282845 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.296654 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.311753 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 198 similar messages]
W20240501 19:57:45.311908 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.312382 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.316432 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.320719 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.331777 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 164 similar messages]
W20240501 19:57:45.338979 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240501 19:57:45.339509 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.346726 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240501 19:57:45.347339 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.347339 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.354830 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.361944 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.366964 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 19:57:45.368816 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:45.399952 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.434185 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:45.449530 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 19:57:45.451984 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:57:45.471217 8770 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:45.471325 8770 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:45.471694 8770 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:45.472580 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:45.472700 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:45.472755 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:45.472923 8770 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:45.480960 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.500484 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.504254 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.511701 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.516177 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.517356 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.525352 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.536743 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:45.580211 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.581004 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.612659 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.646996 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:45.656208 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:57:45.728894 8767 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:45.729038 8767 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:45.729413 8767 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:45.730459 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:45.730561 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:45.730616 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:45.730764 8767 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:45.846964 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.893286 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:45.948943 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:45.989645 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:57:46.045665 8769 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:46.045805 8769 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:46.046198 8769 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:46.047295 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:46.047387 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:46.047436 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:46.047564 8769 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:46.084285 8779 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:46.084393 8779 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:46.084786 8779 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:46.085840 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:46.085922 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:46.085952 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:46.086072 8779 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:46.156111 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:46.181264 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.207643 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.304859 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.344215 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:46.361500 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.414165 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:46.483503 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:57:46.506104 8758 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:46.506215 8758 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:46.506589 8758 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:46.507355 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:46.507484 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:46.507536 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:46.507680 8758 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:46.543437 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:46.555724 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:46.568687 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.660303 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.691213 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:46.716368 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:57:46.759860 8782 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:46.759990 8782 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:46.760357 8782 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:46.761430 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:46.761551 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:46.761595 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:46.761713 8782 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:46.772182 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:46.809255 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.842681 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.855402 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:46.867010 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:46.870499 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:46.936290 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:46.947875 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:46.949689 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.012300 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.042696 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.088294 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:47.094403 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.160037 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.160768 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:47.176201 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.178818 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.212443 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.225957 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.234081 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.289309 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.291225 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.305191 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.319881 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.324954 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.332546 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:47.346800 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.355213 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.355597 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.368573 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:47.386487 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.414692 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.449199 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.493826 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:47.500449 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.511070 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.547853 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.559067 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.559558 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.600862 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.659106 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.660871 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.694154 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.697779 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.705179 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.708173 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:47.723826 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.725512 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.732172 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:47.745313 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.750593 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.795814 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.802989 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.820166 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.833362 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.834751 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.857007 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.868752 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.875306 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.892241 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.906713 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.929248 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:47.950950 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.952104 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.957748 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.972193 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:47.976205 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:48.010231 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.011813 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.033377 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.078174 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:57:48.110494 8767 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:48.110605 8767 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:48.110977 8767 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:48.112015 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:48.112120 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:48.112160 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:48.112272 8767 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:48.142764 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.168107 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.175309 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:48.194335 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.424948 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.446389 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.484573 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:48.551335 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:57:48.565558 8782 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:48.565696 8782 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:48.566076 8782 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:48.567107 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:48.567224 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:48.567263 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:48.567378 8782 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:48.600317 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.601347 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.630059 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:48.634564 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.683669 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.738667 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.770678 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:48.783238 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:57:48.795053 8782 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:48.795161 8782 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:57:48.795516 8782 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:48.796540 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:48.796633 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:48.796664 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:48.796780 8782 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:48.895080 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:48.912940 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:48.913882 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.006708 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.050209 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.134825 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.146970 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.156500 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:49.234375 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.252336 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:49.285293 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.311074 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.344894 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.396127 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.430334 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.444237 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.458688 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:49.510143 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.559832 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.569234 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.573169 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.573518 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:49.612322 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.632033 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.642292 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.678277 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.711159 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.712399 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.733259 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.737715 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.753263 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.800009 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.817080 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.825214 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:49.829373 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.840902 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.869063 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:57:49.869778 8797 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:49.869887 8797 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:49.870252 8797 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:49.871268 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:49.871379 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:49.871438 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:49.871554 8797 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:49.887329 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.890623 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.902150 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.909812 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.942596 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:49.995292 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.001842 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.007249 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:50.025677 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:50.045446 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.049052 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:50.131695 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.160109 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.168524 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.187891 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.219945 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.219990 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.223330 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.223354 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.225289 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.231438 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.241443 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.248453 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.282313 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.286350 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.326601 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 200 similar messages]
W20240501 19:57:50.328755 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.359750 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 149 similar messages]
I20240501 19:57:50.371647 8737 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:50.371788 8737 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:57:50.372272 8737 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:50.372831 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.372895 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:50.373281 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:50.373344 24531 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:50.373461 8737 raft_consensus.cc:2726] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:50.375046 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240501 19:57:50.375633 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.394812 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240501 19:57:50.407474 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240501 19:57:50.423765 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.436725 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.438230 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:50.449259 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.460093 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.461071 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.503858 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:50.509650 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:50.515298 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 19:57:50.523861 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.528582 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.531170 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.542734 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.593726 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.599982 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.601717 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.608286 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.624158 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.635087 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.746366 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:50.979108 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.007663 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.081728 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:51.088388 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:51.201987 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.205195 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.225495 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.233561 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.268078 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.282516 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.355494 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.450001 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:51.460559 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:51.485141 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.535152 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:51.583258 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:51.601348 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.602761 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:51.646019 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:51.713799 8769 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:51.713908 8769 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:51.714291 8769 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:51.715466 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:51.715585 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:51.715634 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:51.715787 8769 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:51.718250 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:51.721056 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.823257 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:51.830839 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.922987 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:51.930465 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:51.957952 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:51.977558 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:52.023299 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:52.049053 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.055992 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:52.062216 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.068181 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.095274 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.159193 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.198302 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.229156 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.246739 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.249136 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.258289 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.267462 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:52.292773 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.294229 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.301815 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.320592 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.321991 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.325999 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.424702 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.448453 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:52.451850 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.485481 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.520905 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.536944 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:52.584607 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.589959 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.598881 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.606842 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.612506 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.638916 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.649152 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:52.661913 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.679139 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:52.684657 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.698104 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.701522 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.735981 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:57:52.740075 8797 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:52.740202 8797 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:52.740622 8797 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:52.741709 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:52.741796 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:52.741842 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:52.741971 8797 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:52.818173 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.829391 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.831394 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.845101 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.851202 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:52.874256 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.880198 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.902855 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.919894 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.929620 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:57:52.933725 8797 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:52.933835 8797 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:52.934219 8797 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:52.935211 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:52.935317 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:52.935350 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:52.935454 8797 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:52.943603 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.946774 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.953194 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.964313 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.969530 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.970286 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:52.979565 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.013594 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.023231 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:53.042748 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.048699 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.054746 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.064009 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.072875 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.093374 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.127661 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:57:53.199151 8768 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:53.199259 8768 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:53.199600 8768 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:53.200587 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:53.200719 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:53.200769 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:53.200871 8768 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:53.222110 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.232307 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:53.345363 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.351327 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.370322 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.446720 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.477861 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.582274 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:53.601768 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:53.626614 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.637161 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.646478 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.651990 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.675626 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:53.820845 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.880928 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:53.940778 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:54.007323 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:54.029263 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:54.102645 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.125797 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:54.174283 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:54.225658 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:54.241919 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:54.244434 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.254506 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.312918 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.474540 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.477129 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.483090 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:54.553280 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:54.561718 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.576831 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.620533 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.631655 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:54.686328 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.706017 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.711128 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.712312 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.721098 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:54.749255 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.749926 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.778206 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.782888 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.793313 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.793999 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.820753 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.897377 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.921447 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.943800 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:54.961966 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.964439 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:54.999156 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.008968 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.018980 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.019986 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.022789 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.033015 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.034198 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.034606 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.057169 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:55.061044 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.062068 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:57:55.093452 8815 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:55.093580 8815 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:57:55.094031 8815 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:57:55.094705 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:57:55.095152 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:55.095319 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:55.095471 8815 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:55.095490 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:55.131117 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.147114 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.157724 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:55.175391 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:57:55.176964 8815 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:55.177096 8815 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:55.177453 8815 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:55.178519 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:55.178618 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:55.178715 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:55.178859 8815 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:55.179343 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.204234 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.211848 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:57:55.223430 8805 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:55.223541 8805 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:57:55.223848 8805 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:55.224915 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:55.225023 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:55.225070 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:55.225234 8805 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:55.256590 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:55.279997 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.300102 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:55.314298 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.328110 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 204 similar messages]
W20240501 19:57:55.362221 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240501 19:57:55.362912 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.390404 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.410306 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 19:57:55.410941 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.429163 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.431423 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:55.445282 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240501 19:57:55.447678 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.467775 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.473264 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:55.480854 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.484606 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.520303 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.546356 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.546540 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.551784 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.560601 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240501 19:57:55.561268 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.598167 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.599426 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 19:57:55.630587 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.666774 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.709246 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.715850 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.730862 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.739470 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.739938 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:55.751005 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:57:55.864244 8828 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:55.864356 8828 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:55.864694 8828 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:55.865643 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:55.865747 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:55.865800 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:55.865932 8828 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:55.882414 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.934273 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:55.942914 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:55.961441 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.100874 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.131978 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.169050 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:56.224778 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:56.299084 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.315444 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.335687 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.370831 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:56.383344 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:56.407833 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.464715 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:56.507562 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.514364 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.518323 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.558223 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:56.706995 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:56.719476 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.780726 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:56.788904 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:56.910943 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:56.916131 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:56.947463 8802 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:56.947579 8802 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:56.947961 8802 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:56.948933 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:56.949033 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:56.949083 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:56.949237 8802 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:56.992622 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:57.010270 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.041781 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.043710 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.051451 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.062650 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:57.070212 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:57:57.101063 8805 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:57:57.101164 8805 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:57.101516 8805 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:57.102479 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:57.102591 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:57.102633 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:57.102746 8805 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:57.113293 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.131219 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.142679 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.150050 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.173609 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.201001 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.204192 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.207887 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.232996 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.267735 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:57.298137 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.312176 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.318506 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:57.354611 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.392597 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.394654 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.395380 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.471989 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:57.474627 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:57.488193 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.489853 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.490123 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.492811 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.510963 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.526135 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.538336 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.551944 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.564496 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.594336 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.618165 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.646905 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.661167 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.675184 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.694296 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.713094 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.733098 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.755048 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.778357 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.779927 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.789862 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.830359 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.847929 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:57.863283 8818 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:57.863443 8818 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:57:57.863842 8818 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:57:57.864295 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.865296 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:57.865407 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:57.865458 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:57.865646 8818 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:57.874182 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.905864 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.923192 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.945580 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.957033 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:57.972301 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:57.985797 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:58.069850 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.097450 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:58.101644 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.109741 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.119660 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.138157 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.144033 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.147148 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.151921 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.173965 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.228834 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.241151 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.294553 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:58.365814 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:58.396227 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.396665 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:58.515107 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.573730 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.640884 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:58.681545 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.695327 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.696484 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:58.707919 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:58.716470 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.800173 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.847929 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.871559 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:58.957063 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:58.992139 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:59.005565 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.076393 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:59.100831 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:59.106528 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.183614 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:57:59.188587 8818 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:59.188717 8818 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:57:59.189112 8818 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:59.190062 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:59.190188 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:59.190232 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:59.190336 8818 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:59.242190 8838 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:59.242307 8838 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:59.242707 8838 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:59.243803 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:59.243899 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:59.243965 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:59.244086 8838 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:59.253000 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:59.300053 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:59.309172 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.320144 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.323817 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:57:59.349633 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:59.432250 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.473052 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.551448 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:59.555378 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.561415 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.575269 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.577270 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.605846 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:59.612262 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.615537 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.642472 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.667377 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.690676 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:57:59.699199 8838 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:57:59.699316 8838 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:57:59.699700 8838 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:57:59.700695 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:59.700812 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:57:59.700870 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:57:59.701009 8838 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:59.717890 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.774091 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.785773 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.819080 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.831130 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.832763 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:59.846909 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.857477 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.895895 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:57:59.908164 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.916680 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.918772 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:59.948371 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:00.061416 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.063174 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.092826 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.103178 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.117873 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.148145 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.153391 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.187883 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.193135 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.210992 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.256033 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.267207 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.271178 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.282230 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.290761 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.292824 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.298491 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.317785 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.354624 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 199 similar messages]
W20240501 19:58:00.355266 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.369367 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.387816 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.400665 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.403142 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240501 19:58:00.420509 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:00.450904 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240501 19:58:00.452658 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 19:58:00.453302 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:00.464123 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.513223 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:00.523347 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.548323 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.596295 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.603894 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 19:58:00.636735 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 19:58:00.643788 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.643788 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.716310 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:00.733246 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.744513 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.753031 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.755214 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.780547 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.792786 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.804097 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.805236 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.850060 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:00.858688 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:00.869772 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:00.946764 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:00.958545 8843 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:00.958679 8843 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:00.959108 8843 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:00.960042 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:00.960233 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:00.960305 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:00.960427 8802 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:58:01.086524 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:01.113705 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.164955 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:01.165978 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:01.234966 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.301313 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.339260 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.378001 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.382899 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.394701 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.404016 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:01.433548 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:01.492609 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.499867 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.561473 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.574543 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.621999 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:01.648667 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:01.695137 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.734242 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:01.747749 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.762341 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:01.824790 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:01.894753 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:01.911460 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:01.961887 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:02.000876 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.020406 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.027815 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.028940 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.044718 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.062337 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:02.118553 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.145880 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.159505 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.163789 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.189688 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.231894 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:02.242964 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:02.266770 8850 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:02.266903 8850 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:58:02.267280 8850 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:02.268646 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:02.268780 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:02.268836 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:02.268960 8843 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:02.286898 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:02.288349 8844 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:02.288480 8844 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:58:02.288947 8844 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:58:02.289727 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:58:02.290009 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:02.290071 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:58:02.290192 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:02.290367 8844 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:02.307562 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.316221 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.318874 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.351475 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.385762 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:02.423959 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.452664 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:02.479482 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.490103 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.491439 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:02.499866 8856 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:02.500006 8856 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:58:02.500432 8856 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:02.501576 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:02.501664 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:02.501706 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:02.501830 8856 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:02.509011 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.519065 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:02.521311 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.561005 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.589321 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.616511 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.626248 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.663703 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.682485 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.701575 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.731173 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.736416 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.749194 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.771826 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.774888 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.816048 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.862776 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.881920 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.887691 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.891630 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:02.910926 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.913434 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.938798 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.949541 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:02.955807 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.002610 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.051390 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:03.062999 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.070581 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.081931 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.097942 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:03.144150 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:03.155004 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.165560 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.170106 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.250191 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.253116 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.279629 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.287683 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.336146 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.411206 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.455018 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:03.544838 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.570613 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:03.572765 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.573786 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:03.588443 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.589582 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.640398 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.642679 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.648814 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:03.669653 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.690941 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:03.725414 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.889956 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.945806 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:03.958048 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:03.990445 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.004232 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:04.076854 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.113606 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:04.156193 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:04.182979 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:04.192246 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:04.228561 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:04.253643 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.258023 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.279085 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:04.286962 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.302533 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.390591 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:04.402537 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:04.415300 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.424436 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.438766 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.445392 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.453402 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.467505 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.484071 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:04.529255 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.656214 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:04.664764 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:04.665484 8862 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:04.665585 8862 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:04.665982 8862 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:04.666957 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:04.667377 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:04.667433 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:04.667659 8862 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:04.697551 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.704768 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.720551 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.730239 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.739511 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:04.771147 8863 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:04.771260 8863 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:04.771621 8863 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:04.773456 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:04.774111 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:04.774168 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:04.774315 8863 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:04.792651 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.793540 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.849745 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:04.896351 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.914491 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.943825 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.969813 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:04.970140 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:05.004664 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.007805 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.011466 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.019778 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.050467 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.136497 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.137051 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.159364 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.164077 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.171973 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.175122 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.196080 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.229836 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.235440 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.239320 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.256078 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:05.277271 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.287338 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.312554 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.320797 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.332582 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:05.394140 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 198 similar messages]
W20240501 19:58:05.418155 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.434763 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.435442 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240501 19:58:05.436336 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.464988 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240501 19:58:05.467999 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.483238 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 168 similar messages]
W20240501 19:58:05.508869 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.576977 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.580610 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.588847 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.605235 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:05.627522 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.636382 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.646983 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 19:58:05.647567 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.651607 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.665966 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 94 similar messages]
I20240501 19:58:05.673691 8866 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:05.673782 8866 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:58:05.674160 8866 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:05.675787 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:05.675892 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:05.675944 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:05.676059 8866 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:05.676621 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.688547 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:05.694671 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.722615 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.735242 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:05.802651 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:05.907634 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:05.961258 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.025300 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.081334 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.139451 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.195662 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.201939 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.208554 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:06.213363 8864 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:06.213467 8864 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:06.213794 8864 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:06.214656 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:06.214761 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:06.214809 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:06.214941 8864 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:06.242944 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:06.266664 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.274268 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.286103 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.287238 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:06.310170 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.330740 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.341434 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.343165 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:06.444746 8870 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:06.444880 8870 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:06.445220 8870 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:06.446226 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:06.446333 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:06.446389 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:06.446552 8870 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:06.508956 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.517930 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.521210 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.532481 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:06.568121 8868 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:06.568269 8868 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:06.568646 8868 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:06.569914 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:06.570047 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:06.570098 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:06.570232 8868 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:58:06.577802 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.593127 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.638006 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.682690 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.704636 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:06.714572 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.766049 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.786260 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.801087 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.813453 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.825812 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:06.829941 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.832136 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:06.835703 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:07.006896 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.031857 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:07.068078 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.112223 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.123637 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.132174 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:07.152053 8871 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:07.152170 8871 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:58:07.152498 8871 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:07.154009 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:07.154151 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:07.154207 24531 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:07.154326 8871 raft_consensus.cc:2726] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:07.172137 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.207578 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.220531 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.234618 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:07.268060 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.282899 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:07.293121 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.298810 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.313745 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.315452 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:07.323956 8874 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:07.324052 8874 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:58:07.324402 8874 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:07.325907 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:07.325981 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:07.326006 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:07.326123 8874 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:07.327916 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:07.341511 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.344982 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.395990 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.422626 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:07.432456 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.454293 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.484058 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.508617 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.529384 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.624954 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.667250 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.675310 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.687199 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.697741 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.729307 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.731660 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.737608 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:07.737814 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.744622 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.744884 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.805001 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.831454 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.868842 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.875918 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.905190 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.909186 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.910717 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:07.951723 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.003676 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:08.069394 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.081763 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.125568 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.141883 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.171983 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.242529 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.262279 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.267735 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:08.277294 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.318188 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:08.330046 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.417886 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.422169 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.473172 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.588220 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:08.614111 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:08.617679 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.660950 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.661386 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.665930 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.704041 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:08.717401 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.791468 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.806475 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.813882 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.829033 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.890252 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:08.938881 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.940775 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:08.941951 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:08.954402 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:08.971261 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:08.975425 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:08.991722 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:09.031421 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:09.032469 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.047705 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:09.127976 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.133713 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:09.142051 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:09.203769 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.233464 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:09.295279 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.320395 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.333912 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.334939 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.355212 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:09.368929 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.440255 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.462100 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.503026 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:09.554248 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.618916 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:09.644181 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.665401 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:09.724037 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.751713 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.829285 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.837360 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.863097 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.865453 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.868376 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:09.889961 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.904891 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.909250 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.917306 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.918315 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.948607 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.962226 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:09.962340 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.966905 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:09.974570 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.055409 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:10.073786 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.088629 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.093096 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.160401 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.172591 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.197599 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.269754 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.281983 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.291240 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.320379 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.376369 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.383095 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.383971 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.413426 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.414089 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.415074 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 201 similar messages]
W20240501 19:58:10.415699 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.439332 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.441565 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.448083 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.457576 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240501 19:58:10.465531 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.479365 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240501 19:58:10.502003 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.520970 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.553539 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:10.617950 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240501 19:58:10.620052 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:10.620404 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.628700 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.639855 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.654459 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 19:58:10.709393 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 19:58:10.709738 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:10.797513 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.817224 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.841696 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.867615 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.969494 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.969676 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:10.979660 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.024910 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.041016 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.047617 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:11.134622 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:11.156773 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.161141 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.224085 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.227375 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.230355 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.247993 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:11.258551 8880 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:11.258669 8880 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:58:11.259027 8880 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:11.260017 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:11.260138 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:11.260190 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:11.260311 8880 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:11.288786 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:11.364574 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:11.372220 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.412225 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.439082 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.498512 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.504827 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.518764 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.519765 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.526151 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:11.541944 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.566439 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:11.603984 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:11.605719 8888 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:11.605806 8888 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:58:11.606187 8888 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:58:11.607507 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:58:11.607630 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:11.607704 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:58:11.607833 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:11.607853 8888 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:11.619729 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.650020 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.666163 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.677521 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.698781 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.746950 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.760627 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.766418 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:11.770299 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.800189 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.826508 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:11.912783 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.062402 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.118041 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:12.148540 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:12.149705 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:12.152015 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:12.156186 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.219848 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.237994 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.253782 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.272755 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.280372 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.284390 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.288215 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.294715 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.331643 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.376507 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.387696 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:12.433842 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.438441 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.450107 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.451762 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:12.495151 8893 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:12.495239 8893 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:12.495544 8893 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:12.496268 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:12.496357 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:12.496399 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:12.496534 8893 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:12.526899 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.551627 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.567106 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:12.604202 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.627434 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.665680 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.669461 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:12.716811 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.735633 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.753286 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.780015 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.780495 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.874846 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.884436 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.910082 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.912427 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.923805 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.926303 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:12.949754 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:12.978785 8894 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:12.978899 8894 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:12.979256 8894 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:12.980120 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:12.980211 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:12.980238 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:12.980381 8894 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:13.006196 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.021119 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.025362 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.030582 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.041612 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.048767 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.048986 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.104723 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:13.108909 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.116873 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.132139 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:13.139158 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:13.162567 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.172535 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.222137 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.229705 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.341408 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.406428 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.456307 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.463547 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.463768 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.476258 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.508041 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.511986 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.539425 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:13.572328 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:13.590003 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.673472 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:13.758940 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:13.795544 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:13.850301 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:13.863591 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:13.935089 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:13.973665 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.011987 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.029024 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.038401 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.060693 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.078392 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.083487 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.095113 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.118638 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:14.124971 8907 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:14.125079 8907 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:14.125437 8907 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:14.126739 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:14.126868 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:14.126931 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:14.127086 8907 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:14.132735 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.152832 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.160454 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.210947 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.233369 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.237215 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.282330 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.286319 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:14.294513 8894 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:14.294597 8894 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:14.294919 8894 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:14.296216 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:14.296327 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:14.296380 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:14.296494 8894 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:58:14.321615 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.327109 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:14.361491 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.369508 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:14.370960 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.454061 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.523324 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.532228 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.538323 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.545670 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.626840 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.652424 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.661684 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.691188 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.701442 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.702047 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.702112 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.702443 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.763634 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.829610 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:14.927294 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.927785 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:14.946934 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:14.984417 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:14.995308 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.042456 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.063117 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.063670 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.081389 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.101625 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.153870 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.167572 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.181109 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.193617 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.217574 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:15.230352 8903 raft_consensus.cc:482] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:15.230468 8903 raft_consensus.cc:504] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } }
I20240501 19:58:15.230811 8903 leader_election.cc:288] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:15.231781 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:15.231891 27229 leader_election.cc:334] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:15.231940 27229 leader_election.cc:302] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:15.232053 8903 raft_consensus.cc:2726] T e19523b5482d4e5a82252f77779afb6d P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:15.252257 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.271122 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.287720 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.296190 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.332594 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.345310 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.357967 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.391709 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:15.428619 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 198 similar messages]
W20240501 19:58:15.442720 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.452842 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.456059 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.463618 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.474265 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.484951 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240501 19:58:15.529410 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:15.538146 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240501 19:58:15.538709 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.565832 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.565878 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.585757 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.609261 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.611230 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.615648 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.626410 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 166 similar messages]
I20240501 19:58:15.626849 8918 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:15.626967 8918 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:15.627368 8918 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:15.628268 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:15.628365 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:15.628408 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:15.628553 8918 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:15.677070 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 19:58:15.694972 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.697661 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:15.700582 8909 raft_consensus.cc:482] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
W20240501 19:58:15.700629 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:15.700681 8909 raft_consensus.cc:504] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:58:15.701053 8909 leader_election.cc:288] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:15.701886 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:15.702032 24793 leader_election.cc:334] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:15.702090 24793 leader_election.cc:302] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:15.702207 8909 raft_consensus.cc:2726] T 92906e09f3474d7987082e0d33304886 P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:15.706331 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.768008 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:15.787264 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240501 19:58:15.841418 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:15.841898 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.865590 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.882539 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.906571 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.986562 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:15.993983 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:16.056077 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.113756 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:16.138654 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.188360 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.238273 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.274322 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.309983 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.344445 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.419569 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:16.422631 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.427390 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:16.511695 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.513023 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.519361 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.548391 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.572661 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.573518 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.598982 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.616039 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:16.619100 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.622574 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.680101 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.703235 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.741391 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.741863 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.792711 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.814070 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.818022 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.851150 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:16.856436 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.907828 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.909649 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.927884 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:16.935381 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:16.947595 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:17.014999 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:17.026235 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:17.068169 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:17.078366 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.132844 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.205163 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.206199 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.211894 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.242930 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:17.251153 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.326274 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:17.331561 8926 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:17.331681 8926 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:58:17.332098 8926 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:17.333025 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:17.333143 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:17.333195 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:17.333343 8926 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:17.337396 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.340910 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.350714 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.380185 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.395221 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.417730 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.434163 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.509127 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:17.511104 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.611701 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.612301 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.653797 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.657327 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.674968 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.676842 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.695653 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.701189 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.746770 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.772429 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.803100 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.836930 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.838673 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:17.839164 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.895217 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.904732 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.908542 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.912264 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.922235 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.934767 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.952297 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.954970 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:17.965119 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:18.017689 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:18.018163 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.045187 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.176330 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.193120 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.201193 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.210614 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.222395 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.260224 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.270287 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.287508 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.348359 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.364439 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:18.379141 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.396648 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.480064 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.488485 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:18.527599 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.626868 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.706282 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:18.713529 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:18.718403 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.801252 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:18.819516 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:18.822937 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:18.838806 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:18.870843 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:18.890612 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:18.911115 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:18.936992 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:18.964376 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:18.970317 8935 raft_consensus.cc:482] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:18.970422 8935 raft_consensus.cc:504] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "9d64cdfaeb2d46b29845160e395fc241" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 38893 } }
I20240501 19:58:18.970777 8935 leader_election.cc:288] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:18.971761 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:18.971868 24793 leader_election.cc:334] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:18.971913 24793 leader_election.cc:302] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d64cdfaeb2d46b29845160e395fc241; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:18.972030 8935 raft_consensus.cc:2726] T afc479e18b5e4efa8a8fa113db3dbf1b P 9d64cdfaeb2d46b29845160e395fc241 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:19.023250 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.083269 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.094893 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.112609 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.116339 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.220027 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.240247 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.244299 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.251214 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.273471 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.283977 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.289566 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.294420 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.317589 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.324919 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.331099 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.336694 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.348071 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:19.354230 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.464486 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:19.467972 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.482053 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.497467 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:19.502512 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.529284 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.587771 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.619832 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.622267 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.632926 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.652333 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:19.687606 8904 raft_consensus.cc:482] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:19.687747 8904 raft_consensus.cc:504] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:19.688200 8904 leader_election.cc:288] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:19.689464 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:19.689625 24924 leader_election.cc:334] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:19.689707 24924 leader_election.cc:302] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:19.689846 8933 raft_consensus.cc:2726] T 4b133f0ed2014a0f99812e4cd8d0e422 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:19.704296 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.718544 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.766942 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:19.776566 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.907109 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.933553 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.952924 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.957155 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:19.995988 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.000087 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:20.036824 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.046376 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.061894 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.104629 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.187361 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.227782 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.236246 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:20.276741 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.291240 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.292124 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.299762 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.327381 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.341516 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.342129 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.408058 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.442808 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 201 similar messages]
W20240501 19:58:20.446688 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.455102 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.461760 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.502724 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.536293 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.541195 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240501 19:58:20.541464 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 156 similar messages]
W20240501 19:58:20.542034 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:20.573757 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.583246 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.588411 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.589468 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.598241 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:20.612515 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:20.615824 8937 raft_consensus.cc:482] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:20.615907 8937 raft_consensus.cc:504] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:58:20.616209 8937 leader_election.cc:288] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:20.617149 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:20.617244 26967 leader_election.cc:334] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:20.617287 26967 leader_election.cc:302] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:20.617439 8937 raft_consensus.cc:2726] T 05cc3b1114a04788821eca2c6343f793 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:20.661727 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240501 19:58:20.676533 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.680089 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 19:58:20.680622 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.733235 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:20.749068 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.756773 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.772691 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.787441 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.799376 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240501 19:58:20.801249 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.825778 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.849854 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.860620 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:20.868943 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.012974 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.040294 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.082031 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.110093 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.165974 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:21.224715 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.275071 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:21.293433 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.314564 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.331123 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:21.347375 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:21.392154 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.429000 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.440400 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.473871 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.496831 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.500083 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.509274 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.525753 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.544230 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.560258 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.577240 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:21.640125 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.697428 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.711650 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.754726 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.756855 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.781039 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:21.832875 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.835094 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.842669 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.882629 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.918161 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.929111 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:21.931869 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.933311 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.938444 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:21.974013 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:22.072667 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.106667 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:22.111341 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.116053 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:22.123601 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.172129 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.189635 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.191112 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.223996 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.250939 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.272600 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.291414 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.292440 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:22.301520 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.317545 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.332222 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.394075 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.410379 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:22.440567 8946 raft_consensus.cc:482] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:22.440681 8946 raft_consensus.cc:504] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:22.441045 8946 leader_election.cc:288] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:22.441995 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:22.442099 24662 leader_election.cc:334] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:22.442147 24662 leader_election.cc:302] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:22.442286 8946 raft_consensus.cc:2726] T b4dabe21c38a4e3fa3c044be1c5ba276 P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:22.473199 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:22.505290 8933 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:22.505460 8933 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:22.505904 8933 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:22.507087 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:22.507208 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:22.507201 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:22.507252 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:22.507382 8933 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:58:22.550403 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.560745 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.585072 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.630748 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.674394 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.680101 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.716187 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.722615 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.775132 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.852809 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.862587 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.877178 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:22.887387 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.918897 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.929384 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.953164 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.974606 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.987890 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:22.995419 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.028255 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:23.045922 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.048386 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.049415 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.050717 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.108824 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.127897 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.181041 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:23.204025 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.223264 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.266260 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.273492 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.281303 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.285135 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.299104 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:23.329573 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.344331 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.376796 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.468014 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.493213 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.496299 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.501475 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:23.574975 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:23.608659 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:23.663399 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:23.696071 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:23.795017 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:23.800526 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.802935 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:23.827322 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.832808 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:23.862370 8952 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:23.862694 8952 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:58:23.863143 8952 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:23.864248 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:23.864332 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:23.864363 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:23.864583 8961 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:23.911887 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:23.919186 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.927215 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:23.970594 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:23.984992 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:23.992746 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:24.004705 8961 raft_consensus.cc:482] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:24.004805 8961 raft_consensus.cc:504] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:24.005138 8961 leader_election.cc:288] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:24.006039 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:24.006155 26967 leader_election.cc:334] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:24.006204 26967 leader_election.cc:302] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:24.006361 8961 raft_consensus.cc:2726] T cdfc4bc6b3b1475e81c2d2b555fcea69 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:58:24.008725 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.009243 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.035702 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.050282 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.094650 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.154286 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.201388 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.248620 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.254741 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.271685 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:24.278478 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.302300 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.305568 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.306627 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.308115 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.355458 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.358736 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.464475 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.525911 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.529740 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.581622 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.592746 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:24.616161 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.699247 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.717916 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.724736 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.741291 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.743464 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.770274 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.878319 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:24.889231 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:24.910650 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.941928 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:24.963011 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.018335 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.029731 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:25.044262 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.053719 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.054239 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.135294 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.144176 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.157388 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.217774 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.223918 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.278331 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.287813 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.293020 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:25.295964 8950 raft_consensus.cc:482] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:25.296066 8950 raft_consensus.cc:504] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "bbf7735f82ca4f65a5170ac7909faa47" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 43399 } }
I20240501 19:58:25.296379 8950 leader_election.cc:288] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:25.297209 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:25.297297 24531 leader_election.cc:334] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:25.297345 24531 leader_election.cc:302] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf7735f82ca4f65a5170ac7909faa47; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:25.297483 8950 raft_consensus.cc:2726] T 8b747487a083433d91191aa79458f51e P bbf7735f82ca4f65a5170ac7909faa47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:25.303017 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.334091 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.348073 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.361665 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.396358 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.403672 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.436523 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:25.440165 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.451676 24531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 204 similar messages]
W20240501 19:58:25.454713 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.554972 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.559254 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240501 19:58:25.559818 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.563889 24793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240501 19:58:25.572238 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.586788 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:25.599597 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:25.610050 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:25.613000 8959 raft_consensus.cc:482] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:25.613085 8959 raft_consensus.cc:504] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "53e34bb09d8e41129a5683079ff63610" member_type: VOTER last_known_addr { host: "127.17.65.7" port: 39797 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } }
I20240501 19:58:25.613384 8959 leader_election.cc:288] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957)
W20240501 19:58:25.614492 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.614674 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:25.614779 27229 leader_election.cc:334] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
I20240501 19:58:25.614822 27229 leader_election.cc:302] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e34bb09d8e41129a5683079ff63610; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:25.614941 8959 raft_consensus.cc:2726] T 37558df60b18476da2e153a4f9171266 P 53e34bb09d8e41129a5683079ff63610 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:25.616998 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.619208 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.660607 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.666560 26967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240501 19:58:25.667241 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.675793 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.727173 24924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 19:58:25.728722 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:25.770180 8961 raft_consensus.cc:482] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:25.770277 8961 raft_consensus.cc:504] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } }
I20240501 19:58:25.770627 8961 leader_election.cc:288] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:25.771567 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:25.771762 26967 leader_election.cc:334] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:25.771818 26967 leader_election.cc:302] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:25.772050 8961 raft_consensus.cc:2726] T 4716f9a3e62a495da309ad14614fceb6 P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:25.789649 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.812872 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.824323 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.825201 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.839512 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.856482 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.884104 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.884987 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 19:58:25.885506 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:25.960635 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:26.000559 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.010808 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.030629 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.151708 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:26.181501 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.210865 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.248203 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.285626 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.307231 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.330801 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.433157 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:26.437390 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.452322 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:26.466320 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:26.521394 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.539081 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.574257 24662 consensus_peers.cc:485] T e75026e7e70548e190f81bf79fb30926 P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.578094 24924 consensus_peers.cc:485] T 53981ad3756d4c54994f7aea973c5cde P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.629310 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.641450 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.646379 24793 consensus_peers.cc:485] T 2c52abe3cfff44f88f41e1e885e5f2c9 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.691498 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.694327 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:26.699345 24662 consensus_peers.cc:485] T cb0bb0d3396a436c878f56d157f30eda P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.704109 26967 consensus_peers.cc:485] T dca5963c18944efe8e5a87cd06d970b6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.728119 24531 consensus_peers.cc:485] T fce9f9c2a5be47d3986e3d7b53570b12 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.757591 24662 consensus_peers.cc:485] T c5b0c2b7abe74eb8b8bf30d8b3cf756d P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.830857 26967 consensus_peers.cc:485] T 2a44d1713369478191bfc30457bf61c1 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.861464 24662 consensus_peers.cc:485] T c76585af218d4ebea389db62aa99a3f7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.880759 24662 consensus_peers.cc:485] T d90582276fb3409b984a4bd96f25e291 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.890672 26967 consensus_peers.cc:485] T 4b48df4b84f24a7fa514f802a5442445 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.903878 24531 consensus_peers.cc:485] T 5110e612369847268a739b03260eec35 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.911082 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:26.915565 24924 consensus_peers.cc:485] T 166bb900935040bd9f7723e822780e6c P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:26.954742 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:27.053822 27229 consensus_peers.cc:485] T 536a3171eab9403eab069fa86c70a8b5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.069077 24793 consensus_peers.cc:485] T 6d2603a318c34256a9fa28979ae41fec P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.143358 24793 consensus_peers.cc:485] T f76202780c824bc0b3debfe8728fc007 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.150578 24793 consensus_peers.cc:485] T ee6a321688af41aa8cd97887b60d4f3a P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.162001 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:27.169137 24531 consensus_peers.cc:485] T 6662293cdf374e019f4197eca35ad8b6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.174355 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.189388 27229 consensus_peers.cc:485] T 433347ce828b4d17a70308c421ab4f1f P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:27.223821 26967 consensus_peers.cc:485] T c53ceef9ab1d4ec593f1dfe7bd8695e5 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:58:27.270007 8976 raft_consensus.cc:482] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:27.270552 8976 raft_consensus.cc:504] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:27.270977 8976 leader_election.cc:288] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:27.272027 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:27.272126 24924 leader_election.cc:334] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:27.272172 24924 leader_election.cc:302] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:27.272307 8976 raft_consensus.cc:2726] T 66b741c6b19f4f1181cdea8a78c816e9 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:27.280575 24531 consensus_peers.cc:485] T caf81a72d0304b8aa611d4c1eab5ac26 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:58:27.303409 24443 maintenance_manager.cc:419] P 547d05e083494037ac958fdcd041e13f: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033375
W20240501 19:58:27.316948 26967 consensus_peers.cc:485] T 54e20f102ed94db78eb4e735b843df24 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:58:27.336130 24442 maintenance_manager.cc:643] P 547d05e083494037ac958fdcd041e13f: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.031s sys 0.000s Metrics: {"bytes_written":109152,"cfile_init":1,"dirs.queue_time_us":211,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":960,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":974,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":305,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":167}
I20240501 19:58:27.336959 24443 maintenance_manager.cc:419] P 547d05e083494037ac958fdcd041e13f: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 37415 bytes on disk
I20240501 19:58:27.337409 24442 maintenance_manager.cc:643] P 547d05e083494037ac958fdcd041e13f: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 19:58:27.341441 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:27.368534 24662 consensus_peers.cc:485] T 3b46d4892fda4ae3b749460bdbb2050e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.418893 24531 consensus_peers.cc:485] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.457173 27229 consensus_peers.cc:485] T 687340001b144f3e93004348ea7f66ab P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.494660 27229 consensus_peers.cc:485] T 14decb125f34452a81547e25a09fb966 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.501886 26967 consensus_peers.cc:485] T 5d80b9d644fb4160950462cdcc895052 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:27.502262 24793 consensus_peers.cc:485] T 2e37eadc7d034f11b68349f255227298 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.516156 24531 consensus_peers.cc:485] T 1647b75a7f8f4a28ae66b85d3fbfeb4c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.518604 27229 consensus_peers.cc:485] T ec85cb24f1c445d9a853cabd2fc41aa0 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.525218 24793 consensus_peers.cc:485] T cfb037c9bf1a4ed080745df687ac4a89 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:58:27.560149 8976 raft_consensus.cc:482] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:27.560274 8976 raft_consensus.cc:504] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:27.560643 8976 leader_election.cc:288] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
W20240501 19:58:27.561581 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:27.561690 24924 leader_election.cc:334] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:27.561738 24924 leader_election.cc:302] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9fd7c49d12b48a7ae73ab09876a0b82; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:27.561924 8976 raft_consensus.cc:2726] T 718c7f2f1f074e3c84534b502a4809a2 P b9fd7c49d12b48a7ae73ab09876a0b82 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:58:27.659193 24662 consensus_peers.cc:485] T a4b240b461b548a087fcabdeb938df1e P 9081292597be4a72a42b216fddcd7971 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.661150 24531 consensus_peers.cc:485] T 244a44d25272441c939d6d69644eaf56 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.677767 24793 consensus_peers.cc:485] T a8020390483c4fd98265cbe7d8012435 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.694003 27229 consensus_peers.cc:485] T 2a075e1cbd5d462b8f140c7e01c8ea9f P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.705668 24531 consensus_peers.cc:485] T 5a37dfc7a4324e2b8c43acb51d07a277 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.715170 27229 consensus_peers.cc:485] T 79f6ba04101645aa9b5e894b9db07bf5 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.728773 24793 consensus_peers.cc:485] T d29130c48f7a435498e854973a090b37 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.733597 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:27.738824 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:27.771134 26967 consensus_peers.cc:485] T 5515207fb0134f8da7f58f4cf730fae7 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.845171 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.857362 24924 consensus_peers.cc:485] T dc23e8b5ad8a4911ae766492fb823806 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:27.948812 26967 consensus_peers.cc:485] T 97f518e4e32644dc8a4af48941f85f55 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:27.965245 8979 raft_consensus.cc:482] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a6e54e86a9949e5970b42e8e32e7561)
I20240501 19:58:27.965344 8979 raft_consensus.cc:504] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } } peers { permanent_uuid: "9081292597be4a72a42b216fddcd7971" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 36299 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "b9fd7c49d12b48a7ae73ab09876a0b82" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45261 } }
I20240501 19:58:27.965921 8979 leader_election.cc:288] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579), 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261)
W20240501 19:58:27.966392 24661 leader_election.cc:341] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9fd7c49d12b48a7ae73ab09876a0b82 (127.17.65.4:45261): Not found: Tablet not found: b4b629378083406b932bfbd5407cac3a
W20240501 19:58:27.967165 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:27.967239 24662 leader_election.cc:302] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9081292597be4a72a42b216fddcd7971; no voters: 29d1a658e2b946e0b412e5fef82db9d9, b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:58:27.967384 8979 raft_consensus.cc:2726] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:58:27.967397 24662 leader_election.cc:334] T b4b629378083406b932bfbd5407cac3a P 9081292597be4a72a42b216fddcd7971 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:28.014869 24531 consensus_peers.cc:485] T 39e7bc7629944b3da3fe8fb1ecae32b1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.020390 27229 consensus_peers.cc:485] T 0ef4152fef8749f0994a952095225134 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.028748 26967 consensus_peers.cc:485] T 49bc415748c44bc6a765ccf11ddd7fea P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.042938 24924 consensus_peers.cc:485] T c35b65e8bb074948b55d675b806b41d8 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.043900 24924 consensus_peers.cc:485] T 783db67a272147ed8752c5d6dba4456e P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.087610 27229 consensus_peers.cc:485] T b04034e366b74575abac4a266d86db93 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.092164 27229 consensus_peers.cc:485] T ea7bbf2e0e5f48ed959195deb548b051 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.118171 27229 consensus_peers.cc:485] T 104d22832ca64b91bf802662ba4ec437 P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.120031 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:28.145169 24924 consensus_peers.cc:485] T 733a59323d4d434c89874d1dab8c9b06 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.157665 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:28.164839 24531 consensus_peers.cc:485] T 99737b8809b345ceb15ca74fc313bf4d P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.177333 27229 consensus_peers.cc:485] T 7eaf6cd60c754ba99747df02a43b1b15 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.192596 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.236619 24531 consensus_peers.cc:485] T a4ade4acdb974d5586bf8bc8a6589f1c P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.244158 26967 consensus_peers.cc:485] T 63b4288ae33d4e81a33648001a8c3f42 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.264012 27229 consensus_peers.cc:485] T fe1f00507b9f48d786a378a15baa4d5b P 53e34bb09d8e41129a5683079ff63610 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.268008 24793 consensus_peers.cc:485] T 92dd4bc514f146bb89083fe8a9315308 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.279326 24531 consensus_peers.cc:485] T c7b14aec624f480189365d112f838635 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.301131 24662 consensus_peers.cc:485] T 8080df897fa94d199a2e9917a734b8d7 P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.309530 24793 consensus_peers.cc:485] T 4a4c18dfded64901bb4645af3dbe2dbc P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.311136 24531 consensus_peers.cc:485] T 71a4aa3f94bf4a3b8d84cae1e1765ac9 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.311297 24924 consensus_peers.cc:485] T ca1d950141e14ca68da6d8b3e8254552 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.368474 26967 consensus_peers.cc:485] T abb4f8ef27a54eea89241362c525553b P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.389163 24924 consensus_peers.cc:485] T 022bb802aa6a4c66a71b7464492360ac P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.391309 24793 consensus_peers.cc:485] T 3b692b5b96534e79a32a8b06b92f32d6 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:28.453893 26967 consensus_peers.cc:485] T a73e21b4daff45a3990876f060497386 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.536060 24531 consensus_peers.cc:462] T 0064e3e3ae1548bba511bd76fb4ec3e1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:28.554953 24531 consensus_peers.cc:485] T f89e81297b66402ea09bdb415fb7f385 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.619316 24793 consensus_peers.cc:462] T 63b6863529174c1ab51bb3cb2827351e P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
W20240501 19:58:28.634200 24924 consensus_peers.cc:485] T c88765fca05c4d31b62db826b51d29c0 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:28.732882 24924 consensus_peers.cc:485] T 99ffec91a9d34152b7bc71051ae59c43 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:28.736372 24531 consensus_peers.cc:485] T 2e30faedce3742e0ba83de21059613d1 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.762903 24793 consensus_peers.cc:485] T 2724041871e04cfdacb91ba5b4026780 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.763779 26967 consensus_peers.cc:485] T d530d173786b4d3996e781e0c677abf4 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.792830 24531 consensus_peers.cc:485] T 2fed0e67f44543d09dbeda7100c7e0d6 P bbf7735f82ca4f65a5170ac7909faa47 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:28.798357 26967 consensus_peers.cc:485] T 40bff70a9c3447c7ab8045568c779c78 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:58:28.840185 27012 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 19:58:28.840348 24707 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:58:28.842970 27267 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 19:58:28.845317 24836 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:58:28.859705 24575 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 19:58:28.873414 24924 consensus_peers.cc:485] T 453d72f3b14c471cbd6fcebe50afa8dd P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:58:28.882905 24966 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 19:58:28.903821 24793 consensus_peers.cc:485] T ac83705168fc418c9672406a78981279 P 9d64cdfaeb2d46b29845160e395fc241 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:28.912400 27229 consensus_peers.cc:485] T 30d30871b80a4baa8c10489b752b0b63 P 53e34bb09d8e41129a5683079ff63610 -> Peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Couldn't send request to peer 29d1a658e2b946e0b412e5fef82db9d9. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:28.937279 24662 consensus_peers.cc:485] T bcbe96677d2140d1aa22870197a9890f P 9081292597be4a72a42b216fddcd7971 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-task3cEqqr/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 45
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.904762
stderr: W20240501 19:56:44.904843 27479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.905120 27479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.909931 27479 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:45.408712 27479 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:56:45.643983 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e48b2299dd574fcab52da0f4ecad97b9: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:45.644110 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table deae6feaf63e416882a15a456246b972: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:45.644234 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 781aeb5e430941fbb20252d74c9d4bc5: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:45.644344 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table db353e9b5d3e45448f4f999f6d406d58: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:45.644388 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6557f4d959fd41d1adb6493b99703660: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:45.644467 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d3b635ed6a7f49d19cd24a7efc6f99d8: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:45.644546 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c03ef128ef544e1bdcb56a4e3d4c2de: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:45.644593 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecee71e04eb7403e978433da58c86ba2: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:45.644631 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e48b2299dd574fcab52da0f4ecad97b9: could not find any suitable replica to move from server 29d1a658e2b946e0b412e5fef82db9d9 to server b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:56:45.644731 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 93e453e991024af88050d72a55aba4c9: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:45.644778 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3b3115cd780e41c3947d6d65be71908c: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:45.644865 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2eb305082d57496294609737a2b6cb47: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:45.644915 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a8f03c8ee2da4d2ba670f1890884cca3: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:45.644994 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c1bf034787b4d7ab64ef1c0ecc34d24: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:45.645056 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5cbd2b1998fa41ae85388accf831149b: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:45.645102 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d222a7ce3414babb32fd3a6232092c0: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:45.645167 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 905ca9f1ddc548a7a616611f708b5be9: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:45.645212 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4ce9bced17984c458d3757f7e32f02a2: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:45.645241 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c031f124af6547ff886da7ae7c7f62b0: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:45.645301 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 44d3fe858f054489b443502e10e226cc: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:45.645331 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c07fdbaa55724ad7aea8c41fd924a7e4: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:45.645361 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2eb305082d57496294609737a2b6cb47: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:45.645390 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a8f03c8ee2da4d2ba670f1890884cca3: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:45.645424 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table db353e9b5d3e45448f4f999f6d406d58: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:45.645457 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6557f4d959fd41d1adb6493b99703660: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:45.645485 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c1bf034787b4d7ab64ef1c0ecc34d24: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:45.645514 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d3b635ed6a7f49d19cd24a7efc6f99d8: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:45.645545 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecee71e04eb7403e978433da58c86ba2: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:45.676561 27479 rebalancer_tool.cc:1074] tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c: '9081292597be4a72a42b216fddcd7971' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:56:45.704741 27479 rebalancer_tool.cc:1074] tablet 66215de525314ea180c164667f4e9f3f: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:45.734369 27479 rebalancer_tool.cc:1074] tablet cb0bb0d3396a436c878f56d157f30eda: 'bbf7735f82ca4f65a5170ac7909faa47' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:56:45.767244 27479 rebalancer_tool.cc:1074] tablet 6d2603a318c34256a9fa28979ae41fec: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:56:45.833814 27479 rebalancer_tool.cc:1074] tablet 433347ce828b4d17a70308c421ab4f1f: '9081292597be4a72a42b216fddcd7971' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:56:45.862135 27479 rebalancer_tool.cc:1074] tablet a4b240b461b548a087fcabdeb938df1e: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:45.889017 27479 rebalancer_tool.cc:1074] tablet d56fe9e75a9c418f817185542ae4c1d9: 'bbf7735f82ca4f65a5170ac7909faa47' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:56:45.890020 27479 rebalancer_tool.cc:1084] tablet afc342897aa343fc930be1d27a2bb641: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '4a6e54e86a9949e5970b42e8e32e7561' move ignored: Not found: afc342897aa343fc930be1d27a2bb641: tablet not found
I20240501 19:56:45.890100 27479 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:46.115154 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5cbd2b1998fa41ae85388accf831149b: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.115275 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 44d3fe858f054489b443502e10e226cc: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:46.115363 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c03ef128ef544e1bdcb56a4e3d4c2de: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:46.115432 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c031f124af6547ff886da7ae7c7f62b0: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:46.115470 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e48b2299dd574fcab52da0f4ecad97b9: could not find any suitable replica to move from server 3b5968cebd0b4294898fdd89b7a5b3a1 to server b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:56:46.115522 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table deae6feaf63e416882a15a456246b972: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.115588 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2eb305082d57496294609737a2b6cb47: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.115679 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d3b635ed6a7f49d19cd24a7efc6f99d8: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.115739 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 781aeb5e430941fbb20252d74c9d4bc5: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:46.115798 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a8f03c8ee2da4d2ba670f1890884cca3: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:46.115844 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d222a7ce3414babb32fd3a6232092c0: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:46.115891 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c1bf034787b4d7ab64ef1c0ecc34d24: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.115936 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table db353e9b5d3e45448f4f999f6d406d58: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:46.115994 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6557f4d959fd41d1adb6493b99703660: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:46.116035 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c07fdbaa55724ad7aea8c41fd924a7e4: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.116096 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 905ca9f1ddc548a7a616611f708b5be9: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:46.116142 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4ce9bced17984c458d3757f7e32f02a2: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:46.116186 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 93e453e991024af88050d72a55aba4c9: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.116227 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3b3115cd780e41c3947d6d65be71908c: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.116300 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c031f124af6547ff886da7ae7c7f62b0: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:46.116350 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5cbd2b1998fa41ae85388accf831149b: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:46.116401 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a8f03c8ee2da4d2ba670f1890884cca3: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:46.116442 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2eb305082d57496294609737a2b6cb47: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:46.116483 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d3b635ed6a7f49d19cd24a7efc6f99d8: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:46.116525 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d222a7ce3414babb32fd3a6232092c0: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:46.116567 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c1bf034787b4d7ab64ef1c0ecc34d24: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:46.161571 27479 rebalancer_tool.cc:1074] tablet 0d91e1c792c749e7883bb996d58d99ea: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:56:46.202368 27479 rebalancer_tool.cc:1074] tablet 718c7f2f1f074e3c84534b502a4809a2: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:56:46.244472 27479 rebalancer_tool.cc:1074] tablet fcfa6701a958416ca481e0349071caa0: 'bbf7735f82ca4f65a5170ac7909faa47' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:46.307063 27479 rebalancer_tool.cc:1074] tablet 09d33aa5ebfd404ca218b069767d216b: '9081292597be4a72a42b216fddcd7971' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:56:46.400285 27479 rebalancer_tool.cc:1074] tablet a1cf6622eed045e7a864b9e397618ecd: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:56:46.435330 27479 rebalancer_tool.cc:1074] tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:56:46.466346 27479 rebalancer_tool.cc:1074] tablet e1141ea148d74ed594effc258ad44dd1: 'bbf7735f82ca4f65a5170ac7909faa47' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:46.504971 27479 rebalancer_tool.cc:1074] tablet f6dbe23d20b34a348938dcc3571020b7: '9081292597be4a72a42b216fddcd7971' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:56:46.542377 27479 rebalancer_tool.cc:1074] tablet 6f1c61cca30c4d878d921c15158a5947: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:56:46.570991 27479 rebalancer_tool.cc:1074] tablet f89e81297b66402ea09bdb415fb7f385: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:56:46.626989 27479 rebalancer_tool.cc:1074] tablet 0f8f81836b9944aa964a8236912c0183: 'bbf7735f82ca4f65a5170ac7909faa47' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:46.660355 27479 rebalancer_tool.cc:1074] tablet e996e96ce69348c09adc99dc7d9c7125: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:56:49.119755 27479 rebalancer_tool.cc:1123] tablet 3f2d49dcdd8f49d48d8a9bd6d10c332c: '9081292597be4a72a42b216fddcd7971' -> '4a6e54e86a9949e5970b42e8e32e7561' move is abandoned: Not found: 3f2d49dcdd8f49d48d8a9bd6d10c332c: tablet not found
I20240501 19:56:49.758330 27479 rebalancer_tool.cc:1123] tablet d56fe9e75a9c418f817185542ae4c1d9: 'bbf7735f82ca4f65a5170ac7909faa47' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move is abandoned: Not found: d56fe9e75a9c418f817185542ae4c1d9: tablet not found
I20240501 19:56:49.963423 27479 rebalancer_tool.cc:1123] tablet 0d91e1c792c749e7883bb996d58d99ea: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move is abandoned: Not found: 0d91e1c792c749e7883bb996d58d99ea: tablet not found
I20240501 19:56:50.145115 27479 rebalancer_tool.cc:1123] tablet a1cf6622eed045e7a864b9e397618ecd: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move is abandoned: Not found: a1cf6622eed045e7a864b9e397618ecd: tablet not found
I20240501 19:56:50.362108 27479 rebalancer_tool.cc:1123] tablet e1141ea148d74ed594effc258ad44dd1: 'bbf7735f82ca4f65a5170ac7909faa47' -> '53e34bb09d8e41129a5683079ff63610' move is abandoned: Not found: e1141ea148d74ed594effc258ad44dd1: tablet not found
I20240501 19:56:50.362974 27479 rebalancer_tool.cc:1123] tablet f6dbe23d20b34a348938dcc3571020b7: '9081292597be4a72a42b216fddcd7971' -> '29d1a658e2b946e0b412e5fef82db9d9' move is abandoned: Not found: f6dbe23d20b34a348938dcc3571020b7: tablet not found
I20240501 19:56:50.367437 27479 rebalancer_tool.cc:1123] tablet 6f1c61cca30c4d878d921c15158a5947: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move is abandoned: Not found: 6f1c61cca30c4d878d921c15158a5947: tablet not found
I20240501 19:56:50.367494 27479 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:50.549746 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c031f124af6547ff886da7ae7c7f62b0: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:50.549829 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d3f59491f47f42698971cb82f7aa32d3: could not find any suitable replica to move from server 53e34bb09d8e41129a5683079ff63610 to server b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:56:50.549854 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d3b635ed6a7f49d19cd24a7efc6f99d8: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:50.549932 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5cbd2b1998fa41ae85388accf831149b: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:50.549961 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c03ef128ef544e1bdcb56a4e3d4c2de: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:50.550035 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 781aeb5e430941fbb20252d74c9d4bc5: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:50.550117 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e48b2299dd574fcab52da0f4ecad97b9: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:50.550148 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table deae6feaf63e416882a15a456246b972: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:50.550222 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a8f03c8ee2da4d2ba670f1890884cca3: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:50.550271 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6557f4d959fd41d1adb6493b99703660: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:50.550298 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0ea2fc512b5434eaba6ba41e6ba01b4: could not find any suitable replica to move from server 4a6e54e86a9949e5970b42e8e32e7561 to server 9d64cdfaeb2d46b29845160e395fc241
W20240501 19:56:50.550328 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 905ca9f1ddc548a7a616611f708b5be9: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:50.550357 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4ce9bced17984c458d3757f7e32f02a2: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:50.550390 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 781aeb5e430941fbb20252d74c9d4bc5: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:50.550426 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d3f59491f47f42698971cb82f7aa32d3: could not find any suitable replica to move from server 3b5968cebd0b4294898fdd89b7a5b3a1 to server 29d1a658e2b946e0b412e5fef82db9d9
I20240501 19:56:50.585891 27479 rebalancer_tool.cc:1074] tablet 51cd33a27a0a4754bbe044b0ff0ef262: '29d1a658e2b946e0b412e5fef82db9d9' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:56:50.617637 27479 rebalancer_tool.cc:1074] tablet 4851f59ff041424bae97a3ecaa9e191b: '29d1a658e2b946e0b412e5fef82db9d9' -> '9081292597be4a72a42b216fddcd7971' move scheduled
I20240501 19:56:50.648327 27479 rebalancer_tool.cc:1074] tablet 23ae3af247c1456e8b75640c25906faf: 'bbf7735f82ca4f65a5170ac7909faa47' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:56:50.673077 27479 rebalancer_tool.cc:1074] tablet 164460e6fbc44fb1a9a12d52fea9254b: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:50.698623 27479 rebalancer_tool.cc:1074] tablet 3b692b5b96534e79a32a8b06b92f32d6: 'bbf7735f82ca4f65a5170ac7909faa47' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:56:50.728274 27479 rebalancer_tool.cc:1074] tablet 92dd4bc514f146bb89083fe8a9315308: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:56:52.314383 27479 rebalancer_tool.cc:1136] tablet a4b240b461b548a087fcabdeb938df1e: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:56:52.513607 27479 rebalancer_tool.cc:1136] tablet 6d2603a318c34256a9fa28979ae41fec: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:56:52.742720 27479 rebalancer_tool.cc:1123] tablet 09d33aa5ebfd404ca218b069767d216b: '9081292597be4a72a42b216fddcd7971' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move is abandoned: Not found: 09d33aa5ebfd404ca218b069767d216b: tablet not found
I20240501 19:56:52.763798 27479 rebalancer_tool.cc:1136] tablet cb0bb0d3396a436c878f56d157f30eda: 'bbf7735f82ca4f65a5170ac7909faa47' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:56:52.771118 27479 rebalancer_tool.cc:1136] tablet fcfa6701a958416ca481e0349071caa0: 'bbf7735f82ca4f65a5170ac7909faa47' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:56:52.779112 27479 rebalancer_tool.cc:1136] tablet 66215de525314ea180c164667f4e9f3f: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:56:52.790421 27479 rebalancer_tool.cc:1136] tablet 0f8f81836b9944aa964a8236912c0183: 'bbf7735f82ca4f65a5170ac7909faa47' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:56:52.795678 27479 rebalancer_tool.cc:1136] tablet 718c7f2f1f074e3c84534b502a4809a2: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:56:52.801286 27479 rebalancer_tool.cc:1136] tablet 71a4aa3f94bf4a3b8d84cae1e1765ac9: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:56:52.825369 27479 rebalancer_tool.cc:1074] tablet 0ed4b95e555a46c5976ca9e084737ea1: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:52.853291 27479 rebalancer_tool.cc:1074] tablet 9acd3c8577694e799d3f67a0f2f6adcc: 'bbf7735f82ca4f65a5170ac7909faa47' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:52.886993 27479 rebalancer_tool.cc:1074] tablet e75026e7e70548e190f81bf79fb30926: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:56:52.912161 27479 rebalancer_tool.cc:1074] tablet 9a6b9bbaf2a0470482f3c5ff98322b90: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:56:52.913746 27479 rebalancer_tool.cc:1084] tablet b6a84bec086a4dbab9e616a0ace7f309: '4a6e54e86a9949e5970b42e8e32e7561' -> '53e34bb09d8e41129a5683079ff63610' move ignored: Not found: b6a84bec086a4dbab9e616a0ace7f309: tablet not found
I20240501 19:56:52.913811 27479 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:52.987707 30392 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalancer_dynamic_table_new_name', partition-key: (<start>), attempt: 1 } failed: table rebalancer_dynamic_table_new_name was deleted: Table deleted at 2024-05-01 19:56:52 UTC
W20240501 19:56:53.177188 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5d329d80c3ab4a1e8f63df8d48a23642: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server bbf7735f82ca4f65a5170ac7909faa47
W20240501 19:56:53.177528 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4ce9bced17984c458d3757f7e32f02a2: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:53.177609 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5d329d80c3ab4a1e8f63df8d48a23642: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:53.177696 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6557f4d959fd41d1adb6493b99703660: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:56:53.177742 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 905ca9f1ddc548a7a616611f708b5be9: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 29d1a658e2b946e0b412e5fef82db9d9
W20240501 19:56:53.177809 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5cbd2b1998fa41ae85388accf831149b: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:53.177848 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c03ef128ef544e1bdcb56a4e3d4c2de: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:56:53.177904 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a8f03c8ee2da4d2ba670f1890884cca3: could not find any suitable replica to move from server bbf7735f82ca4f65a5170ac7909faa47 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:56:53.177938 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5d329d80c3ab4a1e8f63df8d48a23642: could not find any suitable replica to move from server 53e34bb09d8e41129a5683079ff63610 to server b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:56:53.177975 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 781aeb5e430941fbb20252d74c9d4bc5: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 3b5968cebd0b4294898fdd89b7a5b3a1
I20240501 19:56:53.204203 27479 rebalancer_tool.cc:1074] tablet 536a3171eab9403eab069fa86c70a8b5: '9d64cdfaeb2d46b29845160e395fc241' -> '9081292597be4a72a42b216fddcd7971' move scheduled
I20240501 19:56:53.230188 27479 rebalancer_tool.cc:1074] tablet 166bb900935040bd9f7723e822780e6c: '4a6e54e86a9949e5970b42e8e32e7561' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:56:53.268052 27479 rebalancer_tool.cc:1074] tablet 224ae792734e4d1ab3c28e20d82fdf14: '9081292597be4a72a42b216fddcd7971' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:56:53.296124 27479 rebalancer_tool.cc:1074] tablet a4b240b461b548a087fcabdeb938df1e: 'bbf7735f82ca4f65a5170ac7909faa47' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:56:53.324604 27479 rebalancer_tool.cc:1074] tablet afc479e18b5e4efa8a8fa113db3dbf1b: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:56:53.369875 27479 rebalancer_tool.cc:1074] tablet caf81a72d0304b8aa611d4c1eab5ac26: '3b5968cebd0b4294898fdd89b7a5b3a1' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:56:57.286334 27479 rebalancer_tool.cc:1136] tablet 51cd33a27a0a4754bbe044b0ff0ef262: '29d1a658e2b946e0b412e5fef82db9d9' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move completed: OK
I20240501 19:56:57.292680 27479 rebalancer_tool.cc:1136] tablet e996e96ce69348c09adc99dc7d9c7125: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:56:57.298197 27479 rebalancer_tool.cc:1136] tablet f89e81297b66402ea09bdb415fb7f385: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:56:57.304515 27479 rebalancer_tool.cc:1136] tablet 433347ce828b4d17a70308c421ab4f1f: '9081292597be4a72a42b216fddcd7971' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:56:57.313174 27479 rebalancer_tool.cc:1136] tablet 4851f59ff041424bae97a3ecaa9e191b: '29d1a658e2b946e0b412e5fef82db9d9' -> '9081292597be4a72a42b216fddcd7971' move completed: OK
I20240501 19:56:57.360242 27479 rebalancer_tool.cc:1074] tablet 0f1f7372e5ce42b284cf1179c72f578d: '9081292597be4a72a42b216fddcd7971' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:56:57.389206 27479 rebalancer_tool.cc:1074] tablet 2e30faedce3742e0ba83de21059613d1: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:56:57.426146 27479 rebalancer_tool.cc:1074] tablet 4a4c18dfded64901bb4645af3dbe2dbc: '29d1a658e2b946e0b412e5fef82db9d9' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:56:57.458453 27479 rebalancer_tool.cc:1074] tablet 2a44d1713369478191bfc30457bf61c1: '53e34bb09d8e41129a5683079ff63610' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:56:57.510880 27479 rebalancer_tool.cc:1074] tablet 0ef4152fef8749f0994a952095225134: '4a6e54e86a9949e5970b42e8e32e7561' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:00.220798 27479 rebalancer_tool.cc:1136] tablet a4b240b461b548a087fcabdeb938df1e: 'bbf7735f82ca4f65a5170ac7909faa47' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:00.227147 27479 rebalancer_tool.cc:1136] tablet 166bb900935040bd9f7723e822780e6c: '4a6e54e86a9949e5970b42e8e32e7561' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:57:00.233716 27479 rebalancer_tool.cc:1136] tablet 536a3171eab9403eab069fa86c70a8b5: '9d64cdfaeb2d46b29845160e395fc241' -> '9081292597be4a72a42b216fddcd7971' move completed: OK
I20240501 19:57:00.239691 27479 rebalancer_tool.cc:1136] tablet e75026e7e70548e190f81bf79fb30926: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:00.245388 27479 rebalancer_tool.cc:1136] tablet 9acd3c8577694e799d3f67a0f2f6adcc: 'bbf7735f82ca4f65a5170ac7909faa47' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:57:00.251178 27479 rebalancer_tool.cc:1136] tablet 0ed4b95e555a46c5976ca9e084737ea1: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:57:00.256894 27479 rebalancer_tool.cc:1136] tablet 92dd4bc514f146bb89083fe8a9315308: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:57:00.262190 27479 rebalancer_tool.cc:1136] tablet 3b692b5b96534e79a32a8b06b92f32d6: 'bbf7735f82ca4f65a5170ac7909faa47' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:00.267578 27479 rebalancer_tool.cc:1136] tablet 164460e6fbc44fb1a9a12d52fea9254b: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:57:00.503893 27479 rebalancer_tool.cc:1136] tablet 9a6b9bbaf2a0470482f3c5ff98322b90: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'bbf7735f82ca4f65a5170ac7909faa47' move completed: OK
I20240501 19:57:00.509013 27479 rebalancer_tool.cc:1136] tablet 23ae3af247c1456e8b75640c25906faf: 'bbf7735f82ca4f65a5170ac7909faa47' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:57:00.537817 27479 rebalancer_tool.cc:1074] tablet fce9f9c2a5be47d3986e3d7b53570b12: '53e34bb09d8e41129a5683079ff63610' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:00.577730 27479 rebalancer_tool.cc:1074] tablet cfb037c9bf1a4ed080745df687ac4a89: 'bbf7735f82ca4f65a5170ac7909faa47' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:57:00.620615 27479 rebalancer_tool.cc:1074] tablet 53981ad3756d4c54994f7aea973c5cde: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:00.690384 27479 rebalancer_tool.cc:1074] tablet 2c52abe3cfff44f88f41e1e885e5f2c9: '53e34bb09d8e41129a5683079ff63610' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:57:00.717736 27479 rebalancer_tool.cc:1074] tablet 6d2603a318c34256a9fa28979ae41fec: 'bbf7735f82ca4f65a5170ac7909faa47' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:57:00.745904 27479 rebalancer_tool.cc:1074] tablet 5a37dfc7a4324e2b8c43acb51d07a277: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:00.780493 27479 rebalancer_tool.cc:1074] tablet 92906e09f3474d7987082e0d33304886: '53e34bb09d8e41129a5683079ff63610' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:57:00.833918 27479 rebalancer_tool.cc:1074] tablet 8eb1e61bb542415caac5978943e6b3eb: '4a6e54e86a9949e5970b42e8e32e7561' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:00.858208 27479 rebalancer_tool.cc:1074] tablet 0064e3e3ae1548bba511bd76fb4ec3e1: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:00.885149 27479 rebalancer_tool.cc:1074] tablet 2fed0e67f44543d09dbeda7100c7e0d6: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:04.702419 27479 rebalancer_tool.cc:1136] tablet caf81a72d0304b8aa611d4c1eab5ac26: '3b5968cebd0b4294898fdd89b7a5b3a1' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:57:05.312390 27479 rebalancer_tool.cc:1136] tablet afc479e18b5e4efa8a8fa113db3dbf1b: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '9d64cdfaeb2d46b29845160e395fc241' move completed: OK
I20240501 19:57:05.320546 27479 rebalancer_tool.cc:1136] tablet 2a44d1713369478191bfc30457bf61c1: '53e34bb09d8e41129a5683079ff63610' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move completed: OK
I20240501 19:57:05.618661 27479 rebalancer_tool.cc:1136] tablet 0ef4152fef8749f0994a952095225134: '4a6e54e86a9949e5970b42e8e32e7561' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:05.648548 27479 rebalancer_tool.cc:1074] tablet f1cd92f84a944b94b5948123e3e1d7a1: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:57:05.685209 27479 rebalancer_tool.cc:1074] tablet d29130c48f7a435498e854973a090b37: '3b5968cebd0b4294898fdd89b7a5b3a1' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:07.893440 27479 rebalancer_tool.cc:1136] tablet 6d2603a318c34256a9fa28979ae41fec: 'bbf7735f82ca4f65a5170ac7909faa47' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:57:07.900624 27479 rebalancer_tool.cc:1136] tablet 2c52abe3cfff44f88f41e1e885e5f2c9: '53e34bb09d8e41129a5683079ff63610' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move completed: OK
I20240501 19:57:07.906195 27479 rebalancer_tool.cc:1136] tablet 53981ad3756d4c54994f7aea973c5cde: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:07.912878 27479 rebalancer_tool.cc:1136] tablet cfb037c9bf1a4ed080745df687ac4a89: 'bbf7735f82ca4f65a5170ac7909faa47' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:57:07.921663 27479 rebalancer_tool.cc:1136] tablet fce9f9c2a5be47d3986e3d7b53570b12: '53e34bb09d8e41129a5683079ff63610' -> 'bbf7735f82ca4f65a5170ac7909faa47' move completed: OK
I20240501 19:57:08.285881 27479 rebalancer_tool.cc:1136] tablet 2e30faedce3742e0ba83de21059613d1: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:57:08.730881 27479 rebalancer_tool.cc:1136] tablet 0064e3e3ae1548bba511bd76fb4ec3e1: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:08.736084 27479 rebalancer_tool.cc:1136] tablet 8eb1e61bb542415caac5978943e6b3eb: '4a6e54e86a9949e5970b42e8e32e7561' -> 'bbf7735f82ca4f65a5170ac7909faa47' move completed: OK
I20240501 19:57:08.741662 27479 rebalancer_tool.cc:1136] tablet 224ae792734e4d1ab3c28e20d82fdf14: '9081292597be4a72a42b216fddcd7971' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:57:08.788854 27479 rebalancer_tool.cc:1074] tablet 8b747487a083433d91191aa79458f51e: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:08.823763 27479 rebalancer_tool.cc:1074] tablet 98850f18f10c4a57aaaf7a4bd9dce783: '4a6e54e86a9949e5970b42e8e32e7561' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:57:09.827298 27479 rebalancer_tool.cc:1136] tablet 2fed0e67f44543d09dbeda7100c7e0d6: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'bbf7735f82ca4f65a5170ac7909faa47' move completed: OK
I20240501 19:57:09.832899 27479 rebalancer_tool.cc:1136] tablet 92906e09f3474d7987082e0d33304886: '53e34bb09d8e41129a5683079ff63610' -> '9d64cdfaeb2d46b29845160e395fc241' move completed: OK
I20240501 19:57:10.134167 27479 rebalancer_tool.cc:1136] tablet 5a37dfc7a4324e2b8c43acb51d07a277: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:10.143101 27479 rebalancer_tool.cc:1136] tablet 4a4c18dfded64901bb4645af3dbe2dbc: '29d1a658e2b946e0b412e5fef82db9d9' -> '9d64cdfaeb2d46b29845160e395fc241' move completed: OK
I20240501 19:57:10.159579 27479 rebalancer_tool.cc:1136] tablet 0f1f7372e5ce42b284cf1179c72f578d: '9081292597be4a72a42b216fddcd7971' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:57:10.192584 27479 rebalancer_tool.cc:1074] tablet 6ca876a9d51e4096955127e7038e2555: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:57:10.239140 27479 rebalancer_tool.cc:1074] tablet dc23e8b5ad8a4911ae766492fb823806: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:57:10.273882 27479 rebalancer_tool.cc:1074] tablet 783db67a272147ed8752c5d6dba4456e: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:11.377552 27479 rebalancer_tool.cc:1136] tablet f1cd92f84a944b94b5948123e3e1d7a1: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:57:11.686872 27479 rebalancer_tool.cc:1136] tablet 98850f18f10c4a57aaaf7a4bd9dce783: '4a6e54e86a9949e5970b42e8e32e7561' -> '9d64cdfaeb2d46b29845160e395fc241' move completed: OK
I20240501 19:57:11.730883 27479 rebalancer_tool.cc:1074] tablet c3d31791486b4e68a9e5283faadb3f8c: '9d64cdfaeb2d46b29845160e395fc241' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:57:11.786276 27479 rebalancer_tool.cc:1074] tablet 39e7bc7629944b3da3fe8fb1ecae32b1: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:57:12.807348 27479 rebalancer_tool.cc:1136] tablet 8b747487a083433d91191aa79458f51e: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> 'bbf7735f82ca4f65a5170ac7909faa47' move completed: OK
I20240501 19:57:13.160584 27479 rebalancer_tool.cc:1136] tablet d29130c48f7a435498e854973a090b37: '3b5968cebd0b4294898fdd89b7a5b3a1' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:13.166371 27479 rebalancer_tool.cc:1136] tablet dc23e8b5ad8a4911ae766492fb823806: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:57:13.207629 27479 rebalancer_tool.cc:1074] tablet cb0bb0d3396a436c878f56d157f30eda: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:13.816157 27479 rebalancer_tool.cc:1136] tablet 783db67a272147ed8752c5d6dba4456e: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
I20240501 19:57:13.822026 27479 rebalancer_tool.cc:1136] tablet 6ca876a9d51e4096955127e7038e2555: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '9d64cdfaeb2d46b29845160e395fc241' move completed: OK
I20240501 19:57:13.827262 27479 rebalancer_tool.cc:1136] tablet 39e7bc7629944b3da3fe8fb1ecae32b1: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:57:14.150421 27479 rebalancer_tool.cc:1136] tablet c3d31791486b4e68a9e5283faadb3f8c: '9d64cdfaeb2d46b29845160e395fc241' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:57:14.626536 27479 rebalancer_tool.cc:1136] tablet cb0bb0d3396a436c878f56d157f30eda: '9d64cdfaeb2d46b29845160e395fc241' -> '29d1a658e2b946e0b412e5fef82db9d9' move completed: OK
W20240501 19:57:15.270092 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5471c2378ba42119681aafa1ad837ba: could not find any suitable replica to move from server 9081292597be4a72a42b216fddcd7971 to server bbf7735f82ca4f65a5170ac7909faa47
W20240501 19:57:15.270740 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table af59a4587e6e4b38b4931ba4ef49cba5: could not find any suitable replica to move from server 9d64cdfaeb2d46b29845160e395fc241 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:57:15.270800 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c08a0985c3ef425fbe36648030168464: could not find any suitable replica to move from server 53e34bb09d8e41129a5683079ff63610 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:57:15.270879 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5471c2378ba42119681aafa1ad837ba: could not find any suitable replica to move from server 29d1a658e2b946e0b412e5fef82db9d9 to server 3b5968cebd0b4294898fdd89b7a5b3a1
W20240501 19:57:15.271189 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f04654f98149f08331cfb97700b4b1: could not find any suitable replica to move from server 3b5968cebd0b4294898fdd89b7a5b3a1 to server 53e34bb09d8e41129a5683079ff63610
W20240501 19:57:15.271342 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c2451a10f568427881412a162ce73741: could not find any suitable replica to move from server 3b5968cebd0b4294898fdd89b7a5b3a1 to server bbf7735f82ca4f65a5170ac7909faa47
W20240501 19:57:15.271477 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5d329d80c3ab4a1e8f63df8d48a23642: could not find any suitable replica to move from server 53e34bb09d8e41129a5683079ff63610 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:57:15.271539 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5471c2378ba42119681aafa1ad837ba: could not find any suitable replica to move from server b9fd7c49d12b48a7ae73ab09876a0b82 to server 4a6e54e86a9949e5970b42e8e32e7561
W20240501 19:57:15.271656 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 792b7817262f482bb3d6469c2b86c91d: could not find any suitable replica to move from server 4a6e54e86a9949e5970b42e8e32e7561 to server b9fd7c49d12b48a7ae73ab09876a0b82
W20240501 19:57:15.271710 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6052c221112c4cc7ad7bb78b09b40122: could not find any suitable replica to move from server 3b5968cebd0b4294898fdd89b7a5b3a1 to server 9d64cdfaeb2d46b29845160e395fc241
W20240501 19:57:15.271759 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b12e4f4f7fe94a84bc10f1ccdaf184f7: could not find any suitable replica to move from server 4a6e54e86a9949e5970b42e8e32e7561 to server 9d64cdfaeb2d46b29845160e395fc241
W20240501 19:57:15.271847 27479 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a567d922d3d24bb3ad0a2a92c5c09b2f: could not find any suitable replica to move from server 29d1a658e2b946e0b412e5fef82db9d9 to server b9fd7c49d12b48a7ae73ab09876a0b82
I20240501 19:57:15.299641 27479 rebalancer_tool.cc:1074] tablet fb536b9da41e453a9e366a3e20f7ecc2: '9d64cdfaeb2d46b29845160e395fc241' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:57:15.334331 27479 rebalancer_tool.cc:1074] tablet 0f3a2147d24e4fbf92aa1eb88bfb4881: '4a6e54e86a9949e5970b42e8e32e7561' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:57:15.361609 27479 rebalancer_tool.cc:1074] tablet 7959d3c11a6d4c7485c8e030e1d5092b: '9081292597be4a72a42b216fddcd7971' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:57:15.390568 27479 rebalancer_tool.cc:1074] tablet 718c7f2f1f074e3c84534b502a4809a2: 'bbf7735f82ca4f65a5170ac7909faa47' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:15.428121 27479 rebalancer_tool.cc:1074] tablet fcfa6701a958416ca481e0349071caa0: '9d64cdfaeb2d46b29845160e395fc241' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:57:15.460573 27479 rebalancer_tool.cc:1074] tablet cdfc4bc6b3b1475e81c2d2b555fcea69: '53e34bb09d8e41129a5683079ff63610' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:57:15.486712 27479 rebalancer_tool.cc:1074] tablet bb4b1fb143dc410f8f32b68b369b6147: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '9081292597be4a72a42b216fddcd7971' move scheduled
I20240501 19:57:15.511811 27479 rebalancer_tool.cc:1074] tablet c6753d5ac6144c1ba71bc30474f89be7: '29d1a658e2b946e0b412e5fef82db9d9' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:15.538522 27479 rebalancer_tool.cc:1074] tablet 5d80b9d644fb4160950462cdcc895052: '9d64cdfaeb2d46b29845160e395fc241' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
I20240501 19:57:15.565763 27479 rebalancer_tool.cc:1074] tablet 54e20f102ed94db78eb4e735b843df24: '53e34bb09d8e41129a5683079ff63610' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:57:15.619932 27479 rebalancer_tool.cc:1074] tablet 45cabeeee9ad4ec6830a212e93fd2948: '4a6e54e86a9949e5970b42e8e32e7561' -> '9081292597be4a72a42b216fddcd7971' move scheduled
I20240501 19:57:15.645651 27479 rebalancer_tool.cc:1074] tablet 22f2fc10809948cf82331473f18894b3: '29d1a658e2b946e0b412e5fef82db9d9' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:15.673722 27479 rebalancer_tool.cc:1074] tablet 433347ce828b4d17a70308c421ab4f1f: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:57:15.704161 27479 rebalancer_tool.cc:1074] tablet ca1d950141e14ca68da6d8b3e8254552: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:57:15.732249 27479 rebalancer_tool.cc:1074] tablet 733a59323d4d434c89874d1dab8c9b06: '9081292597be4a72a42b216fddcd7971' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:57:15.761847 27479 rebalancer_tool.cc:1074] tablet 38626dc7814f454e85f243fed9607221: '29d1a658e2b946e0b412e5fef82db9d9' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:57:15.803157 27479 rebalancer_tool.cc:1074] tablet 99737b8809b345ceb15ca74fc313bf4d: '53e34bb09d8e41129a5683079ff63610' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:15.836632 27479 rebalancer_tool.cc:1074] tablet 104d22832ca64b91bf802662ba4ec437: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:57:15.868958 27479 rebalancer_tool.cc:1074] tablet f4f099923d754eb791bd4f567f4d2774: '29d1a658e2b946e0b412e5fef82db9d9' -> '9081292597be4a72a42b216fddcd7971' move scheduled
I20240501 19:57:15.916568 27479 rebalancer_tool.cc:1074] tablet 85c67171da5e4b8bbfce0398461ae902: '4a6e54e86a9949e5970b42e8e32e7561' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:23.059290 27479 rebalancer_tool.cc:1136] tablet 104d22832ca64b91bf802662ba4ec437: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move completed: OK
I20240501 19:57:23.066354 27479 rebalancer_tool.cc:1136] tablet 38626dc7814f454e85f243fed9607221: '29d1a658e2b946e0b412e5fef82db9d9' -> '9d64cdfaeb2d46b29845160e395fc241' move completed: OK
I20240501 19:57:23.074741 27479 rebalancer_tool.cc:1136] tablet 733a59323d4d434c89874d1dab8c9b06: '9081292597be4a72a42b216fddcd7971' -> '4a6e54e86a9949e5970b42e8e32e7561' move completed: OK
I20240501 19:57:23.081126 27479 rebalancer_tool.cc:1136] tablet ca1d950141e14ca68da6d8b3e8254552: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move completed: OK
I20240501 19:57:23.086402 27479 rebalancer_tool.cc:1136] tablet 433347ce828b4d17a70308c421ab4f1f: '9d64cdfaeb2d46b29845160e395fc241' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:57:23.091763 27479 rebalancer_tool.cc:1136] tablet 22f2fc10809948cf82331473f18894b3: '29d1a658e2b946e0b412e5fef82db9d9' -> 'bbf7735f82ca4f65a5170ac7909faa47' move completed: OK
I20240501 19:57:23.097735 27479 rebalancer_tool.cc:1136] tablet 45cabeeee9ad4ec6830a212e93fd2948: '4a6e54e86a9949e5970b42e8e32e7561' -> '9081292597be4a72a42b216fddcd7971' move completed: OK
I20240501 19:57:23.103385 27479 rebalancer_tool.cc:1136] tablet 54e20f102ed94db78eb4e735b843df24: '53e34bb09d8e41129a5683079ff63610' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move completed: OK
I20240501 19:57:23.109853 27479 rebalancer_tool.cc:1136] tablet 5d80b9d644fb4160950462cdcc895052: '9d64cdfaeb2d46b29845160e395fc241' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:57:23.116575 27479 rebalancer_tool.cc:1136] tablet c6753d5ac6144c1ba71bc30474f89be7: '29d1a658e2b946e0b412e5fef82db9d9' -> 'bbf7735f82ca4f65a5170ac7909faa47' move completed: OK
I20240501 19:57:23.122468 27479 rebalancer_tool.cc:1136] tablet bb4b1fb143dc410f8f32b68b369b6147: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '9081292597be4a72a42b216fddcd7971' move completed: OK
I20240501 19:57:23.396832 27479 rebalancer_tool.cc:1136] tablet f4f099923d754eb791bd4f567f4d2774: '29d1a658e2b946e0b412e5fef82db9d9' -> '9081292597be4a72a42b216fddcd7971' move completed: OK
I20240501 19:57:23.677894 27479 rebalancer_tool.cc:1136] tablet fcfa6701a958416ca481e0349071caa0: '9d64cdfaeb2d46b29845160e395fc241' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:57:23.683571 27479 rebalancer_tool.cc:1136] tablet 0f3a2147d24e4fbf92aa1eb88bfb4881: '4a6e54e86a9949e5970b42e8e32e7561' -> '53e34bb09d8e41129a5683079ff63610' move completed: OK
I20240501 19:57:23.948849 27479 rebalancer_tool.cc:1136] tablet fb536b9da41e453a9e366a3e20f7ecc2: '9d64cdfaeb2d46b29845160e395fc241' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move completed: OK
I20240501 19:57:23.978155 27479 rebalancer_tool.cc:1074] tablet 244a44d25272441c939d6d69644eaf56: '29d1a658e2b946e0b412e5fef82db9d9' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:57:24.018937 27479 rebalancer_tool.cc:1074] tablet ea7bbf2e0e5f48ed959195deb548b051: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:57:24.051010 27479 rebalancer_tool.cc:1074] tablet a73e21b4daff45a3990876f060497386: '9081292597be4a72a42b216fddcd7971' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:57:24.082856 27479 rebalancer_tool.cc:1074] tablet 2724041871e04cfdacb91ba5b4026780: '3b5968cebd0b4294898fdd89b7a5b3a1' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:57:24.113720 27479 rebalancer_tool.cc:1074] tablet 3b692b5b96534e79a32a8b06b92f32d6: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '53e34bb09d8e41129a5683079ff63610' move scheduled
I20240501 19:57:24.144613 27479 rebalancer_tool.cc:1074] tablet 79f6ba04101645aa9b5e894b9db07bf5: '29d1a658e2b946e0b412e5fef82db9d9' -> '9081292597be4a72a42b216fddcd7971' move scheduled
I20240501 19:57:24.178300 27479 rebalancer_tool.cc:1074] tablet 0064e3e3ae1548bba511bd76fb4ec3e1: '9d64cdfaeb2d46b29845160e395fc241' -> '4a6e54e86a9949e5970b42e8e32e7561' move scheduled
I20240501 19:57:24.227350 27479 rebalancer_tool.cc:1074] tablet d6227a1f28614fbcb1b5a573d125e670: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:24.254431 27479 rebalancer_tool.cc:1074] tablet 63b6863529174c1ab51bb3cb2827351e: 'b9fd7c49d12b48a7ae73ab09876a0b82' -> '29d1a658e2b946e0b412e5fef82db9d9' move scheduled
I20240501 19:57:24.281621 27479 rebalancer_tool.cc:1074] tablet a8020390483c4fd98265cbe7d8012435: '53e34bb09d8e41129a5683079ff63610' -> '9081292597be4a72a42b216fddcd7971' move scheduled
I20240501 19:57:24.311596 27479 rebalancer_tool.cc:1074] tablet 5515207fb0134f8da7f58f4cf730fae7: '4a6e54e86a9949e5970b42e8e32e7561' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:57:24.341576 27479 rebalancer_tool.cc:1074] tablet abb4f8ef27a54eea89241362c525553b: '3b5968cebd0b4294898fdd89b7a5b3a1' -> 'bbf7735f82ca4f65a5170ac7909faa47' move scheduled
I20240501 19:57:24.367554 27479 rebalancer_tool.cc:1074] tablet b4b629378083406b932bfbd5407cac3a: '29d1a658e2b946e0b412e5fef82db9d9' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move scheduled
I20240501 19:57:24.417132 27479 rebalancer_tool.cc:1074] tablet dd1df8a61b1c41368727514e30e263a8: '53e34bb09d8e41129a5683079ff63610' -> '9d64cdfaeb2d46b29845160e395fc241' move scheduled
I20240501 19:57:24.471101 27479 rebalancer_tool.cc:1074] tablet 9acd3c8577694e799d3f67a0f2f6adcc: '9081292597be4a72a42b216fddcd7971' -> '3b5968cebd0b4294898fdd89b7a5b3a1' move scheduled
W20240501 19:57:25.253850 7604 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:25.037641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.65.6:44579 (local address 127.0.0.1:59642)
0501 19:57:25.037776 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:25.037792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:25.037876 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:25.253315 (+215439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":49}
W20240501 19:57:25.266136 7698 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:25.038855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.65.8:35957 (local address 127.0.0.1:55436)
0501 19:57:25.046672 (+ 7817us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:25.046687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:25.046768 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:25.265983 (+219215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":7747,"thread_start_us":75,"threads_started":1}
I20240501 19:57:27.433400 27479 rebalancer_tool.cc:1136] tablet 85c67171da5e4b8bbfce0398461ae902: '4a6e54e86a9949e5970b42e8e32e7561' -> 'bbf7735f82ca4f65a5170ac7909faa47' move completed: OK
I20240501 19:57:28.449798 27479 rebalancer_tool.cc:1123] tablet cdfc4bc6b3b1475e81c2d2b555fcea69: '53e34bb09d8e41129a5683079ff63610' -> '4a6e54e86a9949e5970b42e8e32e7561' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:57:28.658232 27479 rebalancer_tool.cc:1136] tablet 7959d3c11a6d4c7485c8e030e1d5092b: '9081292597be4a72a42b216fddcd7971' -> 'b9fd7c49d12b48a7ae73ab09876a0b82' move completed: OK
I20240501 19:57:28.660676 27479 rebalancer_tool.cc:1123] tablet 718c7f2f1f074e3c84534b502a4809a2: 'bbf7735f82ca4f65a5170ac7909faa47' -> '29d1a658e2b946e0b412e5fef82db9d9' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:28.676420 27479 rebalancer_tool.cc:1084] tablet 97f518e4e32644dc8a4af48941f85f55: '4a6e54e86a9949e5970b42e8e32e7561' -> 'bbf7735f82ca4f65a5170ac7909faa47' move ignored: Timed out: ChangeConfig RPC to 127.17.65.5:45247 timed out after 60.000s (SENT)
I20240501 19:58:28.676522 27479 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): unacceptable health status UNAVAILABLE
I20240501 19:58:28.945653 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 24514
I20240501 19:58:28.955844 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 24645
W20240501 19:58:28.964474 27227 connection.cc:535] client connection to 127.17.65.2:36299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:58:28.964787 24922 connection.cc:535] client connection to 127.17.65.2:36299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:58:28.965617 27227 consensus_peers.cc:485] T 45cabeeee9ad4ec6830a212e93fd2948 P 53e34bb09d8e41129a5683079ff63610 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.2:36299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:58:28.965687 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 24776
W20240501 19:58:28.965693 27227 consensus_peers.cc:485] T c6a7cb30f31b48b49115f8f016c09d82 P 53e34bb09d8e41129a5683079ff63610 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.2:36299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:58:28.966848 24922 consensus_peers.cc:485] T 0f818112217042329c60be020f77df83 P b9fd7c49d12b48a7ae73ab09876a0b82 -> Peer 9081292597be4a72a42b216fddcd7971 (127.17.65.2:36299): Couldn't send request to peer 9081292597be4a72a42b216fddcd7971. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.2:36299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:58:28.972013 26967 consensus_peers.cc:485] T a1e40a2230d149cd91d389c9f2b451f6 P 3b5968cebd0b4294898fdd89b7a5b3a1 -> Peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Couldn't send request to peer 4a6e54e86a9949e5970b42e8e32e7561. Status: Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:58:28.973925 8988 raft_consensus.cc:482] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29d1a658e2b946e0b412e5fef82db9d9)
I20240501 19:58:28.974027 8988 raft_consensus.cc:504] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5968cebd0b4294898fdd89b7a5b3a1" member_type: VOTER last_known_addr { host: "127.17.65.5" port: 45247 } } peers { permanent_uuid: "4a6e54e86a9949e5970b42e8e32e7561" member_type: VOTER last_known_addr { host: "127.17.65.8" port: 35957 } } peers { permanent_uuid: "29d1a658e2b946e0b412e5fef82db9d9" member_type: VOTER last_known_addr { host: "127.17.65.6" port: 44579 } }
I20240501 19:58:28.974465 8988 leader_election.cc:288] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957), 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579)
I20240501 19:58:28.975366 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 24907
W20240501 19:58:28.977003 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a6e54e86a9949e5970b42e8e32e7561 (127.17.65.8:35957): Network error: Client connection negotiation failed: client connection to 127.17.65.8:35957: connect: Connection refused (error 111)
W20240501 19:58:28.977125 26967 leader_election.cc:334] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29d1a658e2b946e0b412e5fef82db9d9 (127.17.65.6:44579): Network error: Client connection negotiation failed: client connection to 127.17.65.6:44579: connect: Connection refused (error 111)
I20240501 19:58:28.977195 26967 leader_election.cc:302] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b5968cebd0b4294898fdd89b7a5b3a1; no voters: 29d1a658e2b946e0b412e5fef82db9d9, 4a6e54e86a9949e5970b42e8e32e7561
I20240501 19:58:28.977319 8988 raft_consensus.cc:2726] T 3ad3e4102cd14025b897ac7da10efe9e P 3b5968cebd0b4294898fdd89b7a5b3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:58:28.984347 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 26950
I20240501 19:58:28.992094 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 27212
I20240501 19:58:29.001332 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 24423
2024-05-01T19:58:29Z chronyd exiting
I20240501 19:58:29.021389 17668 test_util.cc:175] -----------------------------------------------
I20240501 19:58:29.021437 17668 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714593353055615-17668-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (122047 ms)
[----------] 1 test from DDLDuringRebalancingTest (122047 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240501 19:58:29.022063 17668 test_util.cc:268] Using random seed: -385214100
I20240501 19:58:29.022775 17668 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:58:29.022814 17668 ts_itest-base.cc:113] --------------
I20240501 19:58:29.022825 17668 ts_itest-base.cc:114] 8 tablet servers
I20240501 19:58:29.022835 17668 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:58:29.022845 17668 ts_itest-base.cc:116] --------------
2024-05-01T19:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:58:29Z Disabled control of system clock
I20240501 19:58:29.030504 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:45861
--webserver_interface=127.17.65.62
--webserver_port=0
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.65.62:45861
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 19:58:29.127930 9052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:29.128113 9052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:29.128146 9052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:29.130759 9052 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:29.130807 9052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:58:29.130827 9052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:29.130842 9052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:58:29.130857 9052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:58:29.134179 9052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.17.65.62:45861
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.65.62:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.65.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:29.134863 9052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:29.135782 9052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:29.142392 9057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.143023 9060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.144079 9058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:29.144927 9052 server_base.cc:1034] running on GCE node
I20240501 19:58:29.145294 9052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:29.146028 9052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:29.147162 9052 hybrid_clock.cc:648] HybridClock initialized: now 1714593509147153 us; error 22 us; skew 500 ppm
I20240501 19:58:29.148795 9052 webserver.cc:460] Webserver started at http://127.17.65.62:46215/ using document root <none> and password file <none>
I20240501 19:58:29.149283 9052 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:29.149338 9052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:29.149497 9052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:29.150882 9052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.151350 9052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.154266 9052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 19:58:29.156239 9066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.157100 9052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:29.157199 9052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.157269 9052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:29.173648 9052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:29.174175 9052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:29.174309 9052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:29.180596 9052 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.62:45861
I20240501 19:58:29.180614 9119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.62:45861 every 8 connection(s)
I20240501 19:58:29.181488 9052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:58:29.183849 9120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:29.185010 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 9052
I20240501 19:58:29.185127 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:58:29.189167 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc: Bootstrap starting.
I20240501 19:58:29.191184 9120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:29.192036 9120 log.cc:824] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:58:29.193713 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc: No bootstrap required, opened a new log
I20240501 19:58:29.195986 9120 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 45861 } }
I20240501 19:58:29.196143 9120 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:29.196188 9120 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffcfcd00a7d4999a0ff1ebd72b4dadc, State: Initialized, Role: FOLLOWER
I20240501 19:58:29.196494 9120 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 45861 } }
I20240501 19:58:29.196586 9120 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:58:29.196626 9120 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:58:29.196702 9120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:29.197314 9120 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 45861 } }
I20240501 19:58:29.197568 9120 leader_election.cc:302] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffcfcd00a7d4999a0ff1ebd72b4dadc; no voters:
I20240501 19:58:29.197770 9120 leader_election.cc:288] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:58:29.197866 9125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:29.198063 9125 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [term 1 LEADER]: Becoming Leader. State: Replica: fffcfcd00a7d4999a0ff1ebd72b4dadc, State: Running, Role: LEADER
I20240501 19:58:29.198668 9120 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:58:29.198941 9125 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 45861 } }
I20240501 19:58:29.200306 9126 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 45861 } } }
I20240501 19:58:29.200435 9126 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [sys.catalog]: This master's current role is: LEADER
I20240501 19:58:29.200718 9127 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffcfcd00a7d4999a0ff1ebd72b4dadc. Latest consensus state: current_term: 1 leader_uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffcfcd00a7d4999a0ff1ebd72b4dadc" member_type: VOTER last_known_addr { host: "127.17.65.62" port: 45861 } } }
I20240501 19:58:29.200721 9139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:58:29.201058 9127 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc [sys.catalog]: This master's current role is: LEADER
I20240501 19:58:29.203714 9139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:58:29.208473 9139 catalog_manager.cc:1347] Generated new cluster ID: bad6d367dc724c6e82c86138a35c5690
I20240501 19:58:29.208527 9139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:58:29.213714 9139 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:58:29.214440 9139 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:58:29.224120 9139 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fffcfcd00a7d4999a0ff1ebd72b4dadc: Generated new TSK 0
I20240501 19:58:29.224697 9139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:58:29.232342 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.1:0
--local_ip_for_outbound_sockets=127.17.65.1
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:58:29.330988 9144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:29.331246 9144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:29.331313 9144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:29.334663 9144 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:29.334718 9144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:29.334789 9144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.1
I20240501 19:58:29.337860 9144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.65.1
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.9144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.1
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:29.338551 9144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:29.339515 9144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:29.347306 9150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.348004 9152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.348659 9149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:29.349696 9144 server_base.cc:1034] running on GCE node
I20240501 19:58:29.350020 9144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:29.350479 9144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:29.351619 9144 hybrid_clock.cc:648] HybridClock initialized: now 1714593509351604 us; error 32 us; skew 500 ppm
I20240501 19:58:29.353328 9144 webserver.cc:460] Webserver started at http://127.17.65.1:39253/ using document root <none> and password file <none>
I20240501 19:58:29.353787 9144 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:29.353842 9144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:29.353984 9144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:29.355360 9144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.355753 9144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.358587 9144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:58:29.360481 9158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.361382 9144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 19:58:29.361483 9144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.361562 9144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:29.375453 9144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:29.375872 9144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:29.375998 9144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:29.376423 9144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:29.377199 9144 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:29.377246 9144 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.377300 9144 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:29.377338 9144 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.385145 9144 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.1:36523
I20240501 19:58:29.385185 9271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.1:36523 every 8 connection(s)
I20240501 19:58:29.385958 9144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:58:29.387081 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 9144
I20240501 19:58:29.387171 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:58:29.389454 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.2:0
--local_ip_for_outbound_sockets=127.17.65.2
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:29.394749 9272 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:29.394930 9272 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:29.395341 9272 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:29.396749 9077 ts_manager.cc:194] Registered new tserver with Master: 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:29.398375 9077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.1:41341
W20240501 19:58:29.487948 9275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:29.488109 9275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:29.488144 9275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:29.490672 9275 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:29.490717 9275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:29.490772 9275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.2
I20240501 19:58:29.493978 9275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.65.2
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.9275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.2
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:29.494642 9275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:29.495522 9275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:29.503224 9280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.503690 9283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.504392 9281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:29.505487 9275 server_base.cc:1034] running on GCE node
I20240501 19:58:29.505793 9275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:29.506219 9275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:29.507352 9275 hybrid_clock.cc:648] HybridClock initialized: now 1714593509507327 us; error 36 us; skew 500 ppm
I20240501 19:58:29.508895 9275 webserver.cc:460] Webserver started at http://127.17.65.2:45573/ using document root <none> and password file <none>
I20240501 19:58:29.509317 9275 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:29.509361 9275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:29.509492 9275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:29.510815 9275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7b8ab321c9ce4518a42b35047ae72c15"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.511296 9275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7b8ab321c9ce4518a42b35047ae72c15"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.514236 9275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 19:58:29.516098 9289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.516907 9275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:29.517009 9275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7b8ab321c9ce4518a42b35047ae72c15"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.517092 9275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:29.529687 9275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:29.530117 9275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:29.530241 9275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:29.530642 9275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:29.531462 9275 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:29.531512 9275 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.531553 9275 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:29.531585 9275 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.538839 9275 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.2:37481
I20240501 19:58:29.538875 9402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.2:37481 every 8 connection(s)
I20240501 19:58:29.539592 9275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:58:29.545442 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 9275
I20240501 19:58:29.545552 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:58:29.546926 9403 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:29.547133 9403 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:29.547466 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.3:0
--local_ip_for_outbound_sockets=127.17.65.3
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:29.547503 9403 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:29.548307 9077 ts_manager.cc:194] Registered new tserver with Master: 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:29.548954 9077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.2:43563
W20240501 19:58:29.642563 9406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:29.642738 9406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:29.642782 9406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:29.645399 9406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:29.645448 9406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:29.645505 9406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.3
I20240501 19:58:29.648761 9406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.65.3
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.9406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.3
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:29.649420 9406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:29.650300 9406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:29.657899 9411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.658516 9412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.659543 9414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:29.660394 9406 server_base.cc:1034] running on GCE node
I20240501 19:58:29.660715 9406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:29.661165 9406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:29.662315 9406 hybrid_clock.cc:648] HybridClock initialized: now 1714593509662302 us; error 25 us; skew 500 ppm
I20240501 19:58:29.663954 9406 webserver.cc:460] Webserver started at http://127.17.65.3:40849/ using document root <none> and password file <none>
I20240501 19:58:29.664469 9406 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:29.664526 9406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:29.664700 9406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:29.666132 9406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "717a0743a9b741f7909ac2557611bb26"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.666592 9406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "717a0743a9b741f7909ac2557611bb26"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.669679 9406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:58:29.671511 9420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.672335 9406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:29.672446 9406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "717a0743a9b741f7909ac2557611bb26"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.672531 9406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:29.694684 9406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:29.695242 9406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:29.695376 9406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:29.695837 9406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:29.696604 9406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:29.696650 9406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.696700 9406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:29.696728 9406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.705008 9406 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.3:36013
I20240501 19:58:29.705061 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.3:36013 every 8 connection(s)
I20240501 19:58:29.705821 9406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:58:29.713219 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 9406
I20240501 19:58:29.713333 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:58:29.713418 9535 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:29.713603 9535 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:29.714015 9535 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:29.714857 9077 ts_manager.cc:194] Registered new tserver with Master: 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:29.715234 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.4:0
--local_ip_for_outbound_sockets=127.17.65.4
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:29.715442 9077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.3:45405
W20240501 19:58:29.812014 9538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:29.812179 9538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:29.812219 9538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:29.815074 9538 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:29.815122 9538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:29.815182 9538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.4
I20240501 19:58:29.818415 9538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.65.4
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.9538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.4
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:29.819095 9538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:29.820029 9538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:29.827824 9544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.828575 9546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.829437 9543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:29.830272 9538 server_base.cc:1034] running on GCE node
I20240501 19:58:29.830552 9538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:29.830976 9538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:29.832108 9538 hybrid_clock.cc:648] HybridClock initialized: now 1714593509832088 us; error 33 us; skew 500 ppm
I20240501 19:58:29.833709 9538 webserver.cc:460] Webserver started at http://127.17.65.4:43295/ using document root <none> and password file <none>
I20240501 19:58:29.834162 9538 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:29.834226 9538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:29.834357 9538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:29.835789 9538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a32eae53a7ae43a9a0295cc52fa16725"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.836187 9538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a32eae53a7ae43a9a0295cc52fa16725"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.839123 9538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:58:29.841120 9552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.841985 9538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:29.842077 9538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a32eae53a7ae43a9a0295cc52fa16725"
format_stamp: "Formatted at 2024-05-01 19:58:29 on dist-test-slave-703j"
I20240501 19:58:29.842144 9538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:29.862452 9538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:29.862938 9538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:29.863099 9538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:29.863545 9538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:29.864295 9538 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:29.864341 9538 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.864389 9538 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:29.864423 9538 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:29.872339 9538 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.4:45367
I20240501 19:58:29.872364 9665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.4:45367 every 8 connection(s)
I20240501 19:58:29.873131 9538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:58:29.880139 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 9538
I20240501 19:58:29.880261 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:58:29.880561 9666 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:29.880753 9666 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:29.881153 9666 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:29.882036 9077 ts_manager.cc:194] Registered new tserver with Master: a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:29.882333 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.5:0
--local_ip_for_outbound_sockets=127.17.65.5
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:29.882710 9077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.4:40363
W20240501 19:58:29.980129 9669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:29.980309 9669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:29.980347 9669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:29.983126 9669 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:29.983172 9669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:29.983248 9669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.5
I20240501 19:58:29.986454 9669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.65.5
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.9669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.5
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:29.987293 9669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:29.988291 9669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:29.996358 9677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.997032 9675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:29.997870 9674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:29.998744 9669 server_base.cc:1034] running on GCE node
I20240501 19:58:29.999137 9669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:29.999696 9669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:30.000836 9669 hybrid_clock.cc:648] HybridClock initialized: now 1714593510000829 us; error 27 us; skew 500 ppm
I20240501 19:58:30.002652 9669 webserver.cc:460] Webserver started at http://127.17.65.5:39509/ using document root <none> and password file <none>
I20240501 19:58:30.003201 9669 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:30.003257 9669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:30.003446 9669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:30.004992 9669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8ef448fac22b4340ae5d5155c54d6e9f"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.005440 9669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8ef448fac22b4340ae5d5155c54d6e9f"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.008558 9669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:58:30.010490 9683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.011423 9669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:30.011529 9669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8ef448fac22b4340ae5d5155c54d6e9f"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.011606 9669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:30.030222 9669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:30.030783 9669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:30.030928 9669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:30.031433 9669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:30.032232 9669 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:30.032277 9669 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.032322 9669 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:30.032353 9669 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.040014 9669 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.5:36591
I20240501 19:58:30.040041 9796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.5:36591 every 8 connection(s)
I20240501 19:58:30.040936 9669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:58:30.047817 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 9669
I20240501 19:58:30.047945 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:58:30.049062 9797 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:30.049253 9797 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:30.049649 9797 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:30.049841 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.6:0
--local_ip_for_outbound_sockets=127.17.65.6
--webserver_interface=127.17.65.6
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:30.050473 9077 ts_manager.cc:194] Registered new tserver with Master: 8ef448fac22b4340ae5d5155c54d6e9f (127.17.65.5:36591)
I20240501 19:58:30.051043 9077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.5:39279
W20240501 19:58:30.150202 9800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:30.150365 9800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:30.150408 9800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:30.153259 9800 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:30.153311 9800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:30.153371 9800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.6
I20240501 19:58:30.156605 9800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.65.6
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.9800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.6
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:30.157271 9800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:30.158290 9800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:30.166283 9806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:30.167192 9808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:30.167897 9805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:30.168586 9800 server_base.cc:1034] running on GCE node
I20240501 19:58:30.168855 9800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:30.169255 9800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:30.170387 9800 hybrid_clock.cc:648] HybridClock initialized: now 1714593510170365 us; error 32 us; skew 500 ppm
I20240501 19:58:30.171998 9800 webserver.cc:460] Webserver started at http://127.17.65.6:45757/ using document root <none> and password file <none>
I20240501 19:58:30.172452 9800 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:30.172513 9800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:30.172647 9800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:30.174033 9800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "929f8cb9b40f480bbaff000ef35968d8"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.174422 9800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "929f8cb9b40f480bbaff000ef35968d8"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.177448 9800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:58:30.179414 9814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.180358 9800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 19:58:30.180469 9800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "929f8cb9b40f480bbaff000ef35968d8"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.180557 9800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:30.202904 9800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:30.203387 9800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:30.203529 9800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:30.203977 9800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:30.204725 9800 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:30.204767 9800 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.204807 9800 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:30.204836 9800 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.212615 9800 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.6:38277
I20240501 19:58:30.212651 9927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.6:38277 every 8 connection(s)
I20240501 19:58:30.213443 9800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:58:30.215180 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 9800
I20240501 19:58:30.215288 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:58:30.217447 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.7:0
--local_ip_for_outbound_sockets=127.17.65.7
--webserver_interface=127.17.65.7
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:30.222009 9928 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:30.222206 9928 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:30.222671 9928 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:30.223522 9077 ts_manager.cc:194] Registered new tserver with Master: 929f8cb9b40f480bbaff000ef35968d8 (127.17.65.6:38277)
I20240501 19:58:30.224033 9077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.6:38503
W20240501 19:58:30.317054 9931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:30.317230 9931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:30.317296 9931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:30.320171 9931 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:30.320235 9931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:30.320320 9931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.7
I20240501 19:58:30.323962 9931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.65.7
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.9931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.7
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:30.324769 9931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:30.325775 9931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:30.333726 9936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:30.334360 9939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:30.335278 9937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:30.336112 9931 server_base.cc:1034] running on GCE node
I20240501 19:58:30.336403 9931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:30.336882 9931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:30.338037 9931 hybrid_clock.cc:648] HybridClock initialized: now 1714593510338019 us; error 36 us; skew 500 ppm
I20240501 19:58:30.339812 9931 webserver.cc:460] Webserver started at http://127.17.65.7:40049/ using document root <none> and password file <none>
I20240501 19:58:30.340292 9931 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:30.340361 9931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:30.340545 9931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:30.342094 9931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "fc80d0dfc6bd41ecb402d108c9a393ca"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.342571 9931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "fc80d0dfc6bd41ecb402d108c9a393ca"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.345712 9931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 19:58:30.347694 9945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.348590 9931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:30.348696 9931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fc80d0dfc6bd41ecb402d108c9a393ca"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.348770 9931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:30.375985 9931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:30.376529 9931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:30.376670 9931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:30.377130 9931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:30.377918 9931 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:30.377965 9931 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.378021 9931 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:30.378047 9931 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.386227 9931 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.7:39703
I20240501 19:58:30.386274 10058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.7:39703 every 8 connection(s)
I20240501 19:58:30.387120 9931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:58:30.393157 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 9931
I20240501 19:58:30.393280 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:58:30.395378 10059 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:30.395402 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.8:0
--local_ip_for_outbound_sockets=127.17.65.8
--webserver_interface=127.17.65.8
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:30.395563 10059 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:30.395941 10059 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:30.396798 9077 ts_manager.cc:194] Registered new tserver with Master: fc80d0dfc6bd41ecb402d108c9a393ca (127.17.65.7:39703)
I20240501 19:58:30.397542 9077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.7:53033
I20240501 19:58:30.400602 9272 heartbeater.cc:503] Master 127.17.65.62:45861 was elected leader, sending a full tablet report...
W20240501 19:58:30.493525 10062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:30.493700 10062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:30.493757 10062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:30.496444 10062 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:30.496493 10062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:30.496548 10062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.8
I20240501 19:58:30.499933 10062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.17.65.8
--webserver_port=0
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.10062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.8
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:30.500593 10062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:30.501564 10062 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:30.509552 10070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:30.510174 10067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:30.511025 10068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:30.511900 10062 server_base.cc:1034] running on GCE node
I20240501 19:58:30.512255 10062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:30.512759 10062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:30.513900 10062 hybrid_clock.cc:648] HybridClock initialized: now 1714593510513893 us; error 25 us; skew 500 ppm
I20240501 19:58:30.515584 10062 webserver.cc:460] Webserver started at http://127.17.65.8:45667/ using document root <none> and password file <none>
I20240501 19:58:30.516076 10062 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:30.516134 10062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:30.516283 10062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:58:30.517755 10062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d12a830c42564783a28de6d174f48721"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.518172 10062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d12a830c42564783a28de6d174f48721"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.521173 10062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:58:30.523229 10076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.524116 10062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:30.524220 10062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d12a830c42564783a28de6d174f48721"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:30.524289 10062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:30.541052 10062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:30.541522 10062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:30.541661 10062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:30.542088 10062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:30.542845 10062 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:30.542884 10062 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.542923 10062 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:30.542953 10062 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:30.550858 10062 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.8:37673
I20240501 19:58:30.550926 10189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.8:37673 every 8 connection(s)
I20240501 19:58:30.551115 9403 heartbeater.cc:503] Master 127.17.65.62:45861 was elected leader, sending a full tablet report...
I20240501 19:58:30.551851 10062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:58:30.558720 10190 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:30.558869 10190 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:30.559231 10190 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:30.559976 9077 ts_manager.cc:194] Registered new tserver with Master: d12a830c42564783a28de6d174f48721 (127.17.65.8:37673)
I20240501 19:58:30.560446 9077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.8:48797
I20240501 19:58:30.560693 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 10062
I20240501 19:58:30.560818 17668 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:58:30.562765 17668 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:58:30.571309 17668 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:58:30.571393 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 9669
I20240501 19:58:30.577278 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 9800
I20240501 19:58:30.582650 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 9931
I20240501 19:58:30.587647 17668 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3cEqqr/build/debug/bin/kudu with pid 10062
I20240501 19:58:30.717093 9535 heartbeater.cc:503] Master 127.17.65.62:45861 was elected leader, sending a full tablet report...
I20240501 19:58:30.884354 9666 heartbeater.cc:503] Master 127.17.65.62:45861 was elected leader, sending a full tablet report...
I20240501 19:58:34.344101 9077 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:34.362329 9468 tablet_service.cc:1467] Processing CreateTablet for tablet 4198bf0d27614b43bc57396d81a188a1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.363263 9468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4198bf0d27614b43bc57396d81a188a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.363317 9599 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac81d796f0a44b696fba00dd041cb85 (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.363317 9600 tablet_service.cc:1467] Processing CreateTablet for tablet 4198bf0d27614b43bc57396d81a188a1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.363835 9467 tablet_service.cc:1467] Processing CreateTablet for tablet 6123b17fb1944e858db37d008003c94c (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.364110 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6123b17fb1944e858db37d008003c94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.364214 9599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac81d796f0a44b696fba00dd041cb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.366251 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4198bf0d27614b43bc57396d81a188a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.366937 9337 tablet_service.cc:1467] Processing CreateTablet for tablet 6123b17fb1944e858db37d008003c94c (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.367739 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6123b17fb1944e858db37d008003c94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.367986 9206 tablet_service.cc:1467] Processing CreateTablet for tablet 4198bf0d27614b43bc57396d81a188a1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.368505 9205 tablet_service.cc:1467] Processing CreateTablet for tablet 6123b17fb1944e858db37d008003c94c (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.367985 9204 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac81d796f0a44b696fba00dd041cb85 (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.368738 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4198bf0d27614b43bc57396d81a188a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.370501 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6123b17fb1944e858db37d008003c94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.371196 9336 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac81d796f0a44b696fba00dd041cb85 (DEFAULT_TABLE table=rebalance_test_table_0 [id=07e1cd4a02a54b2a943af67436146388]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.371441 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac81d796f0a44b696fba00dd041cb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.371539 10210 tablet_bootstrap.cc:492] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:34.371918 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac81d796f0a44b696fba00dd041cb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.373659 10210 tablet_bootstrap.cc:654] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.374559 10210 log.cc:824] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725: Log is configured to *not* fsync() on all Append() calls
I20240501 19:58:34.376230 10213 tablet_bootstrap.cc:492] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:34.376569 10210 tablet_bootstrap.cc:492] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:34.376739 10210 ts_tablet_manager.cc:1367] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:58:34.378294 10213 tablet_bootstrap.cc:654] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.379015 10213 log.cc:824] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26: Log is configured to *not* fsync() on all Append() calls
I20240501 19:58:34.379354 10211 tablet_bootstrap.cc:492] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:34.380263 10213 tablet_bootstrap.cc:492] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:34.380411 10213 ts_tablet_manager.cc:1367] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:58:34.380400 10210 raft_consensus.cc:348] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.380599 10210 raft_consensus.cc:374] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.380662 10210 raft_consensus.cc:729] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.381115 10211 tablet_bootstrap.cc:654] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.381083 10210 consensus_queue.cc:260] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.381824 10211 log.cc:824] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff: Log is configured to *not* fsync() on all Append() calls
I20240501 19:58:34.381928 10210 ts_tablet_manager.cc:1398] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 19:58:34.382222 10214 tablet_bootstrap.cc:492] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:34.382957 10210 tablet_bootstrap.cc:492] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:34.383329 10211 tablet_bootstrap.cc:492] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:34.383482 10211 ts_tablet_manager.cc:1367] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 19:58:34.383795 10210 tablet_bootstrap.cc:654] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.383805 10213 raft_consensus.cc:348] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.383968 10213 raft_consensus.cc:374] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.384025 10213 raft_consensus.cc:729] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.384449 10213 consensus_queue.cc:260] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.385160 10213 ts_tablet_manager.cc:1398] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:58:34.385478 10213 tablet_bootstrap.cc:492] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:34.386433 10213 tablet_bootstrap.cc:654] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.386914 10211 raft_consensus.cc:348] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.387105 10211 raft_consensus.cc:374] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.387146 10211 raft_consensus.cc:729] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.387511 10211 consensus_queue.cc:260] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.387612 10213 tablet_bootstrap.cc:492] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:34.387703 10213 ts_tablet_manager.cc:1367] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:34.388193 10211 ts_tablet_manager.cc:1398] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240501 19:58:34.388192 10213 raft_consensus.cc:348] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.388291 10213 raft_consensus.cc:374] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.388320 10213 raft_consensus.cc:729] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.388446 10213 consensus_queue.cc:260] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.388700 10213 ts_tablet_manager.cc:1398] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:34.389178 10211 tablet_bootstrap.cc:492] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:34.390002 10211 tablet_bootstrap.cc:654] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
W20240501 19:58:34.390426 9273 tablet.cc:2369] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:58:34.390980 10211 tablet_bootstrap.cc:492] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:34.391075 10211 ts_tablet_manager.cc:1367] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:58:34.391486 10211 raft_consensus.cc:348] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.391589 10211 raft_consensus.cc:374] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.391621 10211 raft_consensus.cc:729] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.391754 10211 consensus_queue.cc:260] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.392040 10211 ts_tablet_manager.cc:1398] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:34.392196 10211 tablet_bootstrap.cc:492] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:34.392971 10211 tablet_bootstrap.cc:654] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.394107 10211 tablet_bootstrap.cc:492] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:34.394196 10211 ts_tablet_manager.cc:1367] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:34.394349 10210 tablet_bootstrap.cc:492] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:34.394428 10210 ts_tablet_manager.cc:1367] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 19:58:34.394603 10211 raft_consensus.cc:348] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.394682 10211 raft_consensus.cc:374] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.394704 10211 raft_consensus.cc:729] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.394799 10211 consensus_queue.cc:260] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.394990 10211 ts_tablet_manager.cc:1398] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:34.395512 10210 raft_consensus.cc:348] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.395614 10210 raft_consensus.cc:374] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.395661 10210 raft_consensus.cc:729] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.396240 10210 consensus_queue.cc:260] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.396484 10210 ts_tablet_manager.cc:1398] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:34.396801 10214 tablet_bootstrap.cc:654] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.398002 10214 log.cc:824] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15: Log is configured to *not* fsync() on all Append() calls
I20240501 19:58:34.399513 10214 tablet_bootstrap.cc:492] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:34.399700 10214 ts_tablet_manager.cc:1367] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 19:58:34.402436 10214 raft_consensus.cc:348] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.402593 10214 raft_consensus.cc:374] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.402643 10214 raft_consensus.cc:729] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.402998 10214 consensus_queue.cc:260] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.403723 10214 ts_tablet_manager.cc:1398] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:58:34.403937 10214 tablet_bootstrap.cc:492] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:34.404770 10214 tablet_bootstrap.cc:654] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.405550 10216 raft_consensus.cc:482] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:34.405676 10216 raft_consensus.cc:504] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.405781 10214 tablet_bootstrap.cc:492] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:34.405833 10214 ts_tablet_manager.cc:1367] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:34.406179 10214 raft_consensus.cc:348] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.406266 10214 raft_consensus.cc:374] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.406301 10214 raft_consensus.cc:729] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.406414 10214 consensus_queue.cc:260] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.406698 10214 ts_tablet_manager.cc:1398] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:34.406766 10216 leader_election.cc:288] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:34.410687 9489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4198bf0d27614b43bc57396d81a188a1" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:34.410882 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4198bf0d27614b43bc57396d81a188a1" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:34.410905 9489 raft_consensus.cc:2454] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 0.
I20240501 19:58:34.411099 9226 raft_consensus.cc:2454] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 0.
I20240501 19:58:34.411283 9556 leader_election.cc:302] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:34.411501 10216 raft_consensus.cc:2781] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:34.411569 10216 raft_consensus.cc:482] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:34.411617 10216 raft_consensus.cc:3037] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.412366 10216 raft_consensus.cc:504] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.412647 10216 leader_election.cc:288] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 election: Requested vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:34.412916 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4198bf0d27614b43bc57396d81a188a1" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:34.412945 9489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4198bf0d27614b43bc57396d81a188a1" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:34.413018 9489 raft_consensus.cc:3037] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.413019 9226 raft_consensus.cc:3037] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.413723 9489 raft_consensus.cc:2454] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 1.
I20240501 19:58:34.413942 9556 leader_election.cc:302] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:34.414081 10216 raft_consensus.cc:2781] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:34.414183 9226 raft_consensus.cc:2454] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 1.
I20240501 19:58:34.414229 10216 raft_consensus.cc:686] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 LEADER]: Becoming Leader. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Running, Role: LEADER
I20240501 19:58:34.414505 10216 consensus_queue.cc:237] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.416704 9083 catalog_manager.cc:5484] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 reported cstate change: term changed from 0 to 1, leader changed from <none> to a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "a32eae53a7ae43a9a0295cc52fa16725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } }
W20240501 19:58:34.460306 9536 tablet.cc:2369] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:58:34.544059 9404 tablet.cc:2369] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:58:34.548225 10226 raft_consensus.cc:482] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:34.548398 10226 raft_consensus.cc:504] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.549345 10226 leader_election.cc:288] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:34.553114 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac81d796f0a44b696fba00dd041cb85" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:34.553149 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac81d796f0a44b696fba00dd041cb85" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:34.553262 9226 raft_consensus.cc:2454] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:34.553318 9620 raft_consensus.cc:2454] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:34.553627 9291 leader_election.cc:302] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:34.553880 10226 raft_consensus.cc:2781] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:34.553964 10226 raft_consensus.cc:482] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:34.554021 10226 raft_consensus.cc:3037] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.554919 10226 raft_consensus.cc:504] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.555291 10226 leader_election.cc:288] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:34.555593 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac81d796f0a44b696fba00dd041cb85" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:34.555721 9620 raft_consensus.cc:3037] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.555692 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac81d796f0a44b696fba00dd041cb85" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:34.555791 9226 raft_consensus.cc:3037] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.556505 9226 raft_consensus.cc:2454] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:34.556727 9291 leader_election.cc:302] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:34.556918 9620 raft_consensus.cc:2454] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:34.556980 10226 raft_consensus.cc:2781] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:34.557165 10226 raft_consensus.cc:686] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:34.557454 10226 consensus_queue.cc:237] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.560483 9083 catalog_manager.cc:5484] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } }
I20240501 19:58:34.573422 10226 raft_consensus.cc:482] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:34.573501 10226 raft_consensus.cc:504] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.573858 10226 leader_election.cc:288] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:34.574136 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6123b17fb1944e858db37d008003c94c" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:34.574262 9226 raft_consensus.cc:2454] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:34.574501 9291 leader_election.cc:302] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:34.574620 10226 raft_consensus.cc:2781] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:34.574664 10226 raft_consensus.cc:482] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:34.574719 10226 raft_consensus.cc:3037] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.575610 10226 raft_consensus.cc:504] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.575934 10226 leader_election.cc:288] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:34.576192 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6123b17fb1944e858db37d008003c94c" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:34.576301 9226 raft_consensus.cc:3037] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.577075 9226 raft_consensus.cc:2454] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:34.577296 9291 leader_election.cc:302] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:34.577323 9489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6123b17fb1944e858db37d008003c94c" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:34.577452 9489 raft_consensus.cc:2454] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:34.577417 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6123b17fb1944e858db37d008003c94c" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:34.577625 9488 raft_consensus.cc:3037] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.577723 10226 raft_consensus.cc:2781] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:34.577786 10226 raft_consensus.cc:686] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:34.577922 10226 consensus_queue.cc:237] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.578554 9488 raft_consensus.cc:2454] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:34.579226 9083 catalog_manager.cc:5484] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
W20240501 19:58:34.627336 9667 tablet.cc:2369] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:58:34.683290 10237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:34.683576 10237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:34.686465 10237 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:58:34.771453 9226 raft_consensus.cc:1261] T 8ac81d796f0a44b696fba00dd041cb85 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:34.771728 9225 raft_consensus.cc:1261] T 6123b17fb1944e858db37d008003c94c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:34.772161 9620 raft_consensus.cc:1261] T 8ac81d796f0a44b696fba00dd041cb85 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:34.772583 10226 consensus_queue.cc:1035] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:34.772667 9225 raft_consensus.cc:1261] T 4198bf0d27614b43bc57396d81a188a1 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer a32eae53a7ae43a9a0295cc52fa16725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:34.772792 9488 raft_consensus.cc:1261] T 6123b17fb1944e858db37d008003c94c P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:34.773005 10226 consensus_queue.cc:1035] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:34.773068 10230 consensus_queue.cc:1035] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:34.773648 10234 consensus_queue.cc:1035] T 6123b17fb1944e858db37d008003c94c P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:34.775350 10234 consensus_queue.cc:1035] T 8ac81d796f0a44b696fba00dd041cb85 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:34.781526 9488 raft_consensus.cc:1261] T 4198bf0d27614b43bc57396d81a188a1 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer a32eae53a7ae43a9a0295cc52fa16725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:34.781936 10230 consensus_queue.cc:1035] T 4198bf0d27614b43bc57396d81a188a1 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:34.785188 10266 mvcc.cc:205] Tried to move back new op lower bound from 7022975036522913792 to 7022975035043721216. Current Snapshot: MvccSnapshot[applied={T|T < 7022975036501041152}]
I20240501 19:58:34.785660 10270 mvcc.cc:205] Tried to move back new op lower bound from 7022975036501041152 to 7022975035043721216. Current Snapshot: MvccSnapshot[applied={T|T < 7022975036501041152}]
I20240501 19:58:34.786602 10265 mvcc.cc:205] Tried to move back new op lower bound from 7022975036496875520 to 7022975035712114688. Current Snapshot: MvccSnapshot[applied={T|T < 7022975036496875520}]
I20240501 19:58:34.795439 10269 mvcc.cc:205] Tried to move back new op lower bound from 7022975036496875520 to 7022975035712114688. Current Snapshot: MvccSnapshot[applied={T|T < 7022975036496875520}]
INSERT report
rows total: 2000
time total: 107.455 ms
time per row: 0.0537277 ms
I20240501 19:58:34.835942 9077 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:34.840745 9468 tablet_service.cc:1467] Processing CreateTablet for tablet cfdecd04308f42659df60c13bf56cb7f (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.840754 9600 tablet_service.cc:1467] Processing CreateTablet for tablet cfdecd04308f42659df60c13bf56cb7f (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.841027 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdecd04308f42659df60c13bf56cb7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.841029 9468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdecd04308f42659df60c13bf56cb7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.841161 9467 tablet_service.cc:1467] Processing CreateTablet for tablet c3633f6cb6dc4bf3ae9871ae89048989 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.841378 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3633f6cb6dc4bf3ae9871ae89048989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.841746 9466 tablet_service.cc:1467] Processing CreateTablet for tablet a0b4daae580844ae828eb73355cbe6d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.841990 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b4daae580844ae828eb73355cbe6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.843175 10213 tablet_bootstrap.cc:492] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:34.843356 9337 tablet_service.cc:1467] Processing CreateTablet for tablet cfdecd04308f42659df60c13bf56cb7f (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.843559 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdecd04308f42659df60c13bf56cb7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.844059 10213 tablet_bootstrap.cc:654] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.844430 9336 tablet_service.cc:1467] Processing CreateTablet for tablet c3633f6cb6dc4bf3ae9871ae89048989 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.844656 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3633f6cb6dc4bf3ae9871ae89048989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.844827 9335 tablet_service.cc:1467] Processing CreateTablet for tablet a0b4daae580844ae828eb73355cbe6d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.845064 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b4daae580844ae828eb73355cbe6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.845537 10210 tablet_bootstrap.cc:492] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:34.846210 9205 tablet_service.cc:1467] Processing CreateTablet for tablet a0b4daae580844ae828eb73355cbe6d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.846376 10210 tablet_bootstrap.cc:654] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.846479 10214 tablet_bootstrap.cc:492] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:34.846508 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b4daae580844ae828eb73355cbe6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.847445 10214 tablet_bootstrap.cc:654] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.847656 9600 tablet_service.cc:1467] Processing CreateTablet for tablet c3633f6cb6dc4bf3ae9871ae89048989 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c8068f07647d417b89d6b2cca6cc902e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:34.847898 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3633f6cb6dc4bf3ae9871ae89048989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:34.848359 10213 tablet_bootstrap.cc:492] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:34.848433 10213 ts_tablet_manager.cc:1367] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:58:34.848506 10214 tablet_bootstrap.cc:492] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:34.848577 10214 ts_tablet_manager.cc:1367] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:34.848876 10213 raft_consensus.cc:348] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.848978 10213 raft_consensus.cc:374] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.849009 10213 raft_consensus.cc:729] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.849051 10211 tablet_bootstrap.cc:492] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:34.849023 10214 raft_consensus.cc:348] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.849110 10214 raft_consensus.cc:374] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.849138 10214 raft_consensus.cc:729] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.849115 10213 consensus_queue.cc:260] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.849242 10214 consensus_queue.cc:260] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.849349 10213 ts_tablet_manager.cc:1398] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:34.849488 10214 ts_tablet_manager.cc:1398] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:34.849505 10213 tablet_bootstrap.cc:492] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:34.849659 10214 tablet_bootstrap.cc:492] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:34.849834 10211 tablet_bootstrap.cc:654] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.850365 10213 tablet_bootstrap.cc:654] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.850417 10214 tablet_bootstrap.cc:654] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.851258 10213 tablet_bootstrap.cc:492] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:34.851333 10213 ts_tablet_manager.cc:1367] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:34.851734 10213 raft_consensus.cc:348] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.851856 10213 raft_consensus.cc:374] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.851898 10213 raft_consensus.cc:729] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.852020 10213 consensus_queue.cc:260] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.852272 10213 ts_tablet_manager.cc:1398] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:34.852293 10210 tablet_bootstrap.cc:492] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:34.852356 10210 ts_tablet_manager.cc:1367] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:58:34.852381 10214 tablet_bootstrap.cc:492] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:34.852408 10213 tablet_bootstrap.cc:492] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:34.852450 10214 ts_tablet_manager.cc:1367] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:58:34.852770 10210 raft_consensus.cc:348] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.852867 10210 raft_consensus.cc:374] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.852834 10214 raft_consensus.cc:348] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.852903 10214 raft_consensus.cc:374] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.852897 10210 raft_consensus.cc:729] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.852923 10214 raft_consensus.cc:729] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.853029 10214 consensus_queue.cc:260] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.853026 10210 consensus_queue.cc:260] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.853209 10213 tablet_bootstrap.cc:654] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.853252 10214 ts_tablet_manager.cc:1398] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:34.853286 10210 ts_tablet_manager.cc:1398] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:34.853379 10214 tablet_bootstrap.cc:492] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:34.853464 10210 tablet_bootstrap.cc:492] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:34.854192 10214 tablet_bootstrap.cc:654] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.854879 10210 tablet_bootstrap.cc:654] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:34.855744 10214 tablet_bootstrap.cc:492] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:34.855757 10211 tablet_bootstrap.cc:492] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:34.855821 10214 ts_tablet_manager.cc:1367] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:34.855821 10211 ts_tablet_manager.cc:1367] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:58:34.856199 10214 raft_consensus.cc:348] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.856199 10211 raft_consensus.cc:348] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.856281 10211 raft_consensus.cc:374] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.856280 10214 raft_consensus.cc:374] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.856312 10211 raft_consensus.cc:729] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.856312 10214 raft_consensus.cc:729] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.856417 10211 consensus_queue.cc:260] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.856426 10214 consensus_queue.cc:260] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.856674 10214 ts_tablet_manager.cc:1398] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:34.856673 10211 ts_tablet_manager.cc:1398] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:34.858230 10213 tablet_bootstrap.cc:492] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:34.858314 10213 ts_tablet_manager.cc:1367] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:58:34.858464 10210 tablet_bootstrap.cc:492] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:34.858536 10210 ts_tablet_manager.cc:1367] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:58:34.858695 10213 raft_consensus.cc:348] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.858806 10213 raft_consensus.cc:374] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.858830 10213 raft_consensus.cc:729] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.858889 10210 raft_consensus.cc:348] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.858932 10213 consensus_queue.cc:260] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.858981 10210 raft_consensus.cc:374] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:34.859017 10210 raft_consensus.cc:729] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:34.859189 10213 ts_tablet_manager.cc:1398] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:34.859251 10210 consensus_queue.cc:260] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:34.859544 10210 ts_tablet_manager.cc:1398] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:34.904290 10288 raft_consensus.cc:482] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:34.904410 10288 raft_consensus.cc:504] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.904763 10288 leader_election.cc:288] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:34.905125 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdecd04308f42659df60c13bf56cb7f" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:34.905125 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdecd04308f42659df60c13bf56cb7f" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:34.905262 9620 raft_consensus.cc:2454] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:34.905262 9488 raft_consensus.cc:2454] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:34.905545 9293 leader_election.cc:302] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:34.905700 10288 raft_consensus.cc:2781] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:34.905754 10288 raft_consensus.cc:482] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:34.905795 10288 raft_consensus.cc:3037] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.906570 10288 raft_consensus.cc:504] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.906867 10288 leader_election.cc:288] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:34.907167 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdecd04308f42659df60c13bf56cb7f" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:34.907210 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdecd04308f42659df60c13bf56cb7f" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:34.907266 9620 raft_consensus.cc:3037] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.907341 9488 raft_consensus.cc:3037] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.908052 9620 raft_consensus.cc:2454] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:34.908113 9488 raft_consensus.cc:2454] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:34.908277 9291 leader_election.cc:302] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:34.908425 10288 raft_consensus.cc:2781] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:34.908501 10288 raft_consensus.cc:686] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:34.908634 10288 consensus_queue.cc:237] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:34.909981 9078 catalog_manager.cc:5484] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } }
I20240501 19:58:34.910451 10284 raft_consensus.cc:482] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:34.910543 10284 raft_consensus.cc:504] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.911584 10284 leader_election.cc:288] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:34.915462 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b4daae580844ae828eb73355cbe6d0" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:34.915663 9226 raft_consensus.cc:2454] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:34.915794 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b4daae580844ae828eb73355cbe6d0" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:34.915979 9357 raft_consensus.cc:2454] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:34.915990 9422 leader_election.cc:302] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:34.916218 10284 raft_consensus.cc:2781] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:34.916298 10284 raft_consensus.cc:482] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:34.916363 10284 raft_consensus.cc:3037] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.917035 10284 raft_consensus.cc:504] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.917291 10284 leader_election.cc:288] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:34.917570 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b4daae580844ae828eb73355cbe6d0" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:34.917570 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b4daae580844ae828eb73355cbe6d0" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:34.917673 9226 raft_consensus.cc:3037] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.917672 9357 raft_consensus.cc:3037] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:34.918509 9226 raft_consensus.cc:2454] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:34.918509 9357 raft_consensus.cc:2454] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:34.918740 9422 leader_election.cc:302] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:34.918879 10284 raft_consensus.cc:2781] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:34.918957 10284 raft_consensus.cc:686] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:34.919267 10284 consensus_queue.cc:237] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:34.921013 9078 catalog_manager.cc:5484] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
I20240501 19:58:35.104332 10284 raft_consensus.cc:482] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:35.104470 10284 raft_consensus.cc:504] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:35.104921 10284 leader_election.cc:288] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:35.105352 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3633f6cb6dc4bf3ae9871ae89048989" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:35.105531 9357 raft_consensus.cc:2454] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:35.105847 9421 leader_election.cc:302] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:35.106025 10284 raft_consensus.cc:2781] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:35.106101 10284 raft_consensus.cc:482] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:35.106139 10284 raft_consensus.cc:3037] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.107180 10284 raft_consensus.cc:504] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:35.107491 10284 leader_election.cc:288] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:35.107847 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3633f6cb6dc4bf3ae9871ae89048989" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:35.107976 9357 raft_consensus.cc:3037] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.108733 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3633f6cb6dc4bf3ae9871ae89048989" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:35.108822 9357 raft_consensus.cc:2454] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:35.108842 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3633f6cb6dc4bf3ae9871ae89048989" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:35.108863 9620 raft_consensus.cc:2454] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:35.109130 9421 leader_election.cc:302] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:35.109277 10284 raft_consensus.cc:2781] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:35.109359 10284 raft_consensus.cc:686] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:35.109499 10284 consensus_queue.cc:237] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:35.111018 9078 catalog_manager.cc:5484] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } }
W20240501 19:58:35.237020 10312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:35.237213 10312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:35.239894 10312 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:58:35.324841 9619 raft_consensus.cc:3037] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.325979 9619 raft_consensus.cc:1261] T c3633f6cb6dc4bf3ae9871ae89048989 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.326392 10219 consensus_queue.cc:1035] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.326756 9357 raft_consensus.cc:1261] T c3633f6cb6dc4bf3ae9871ae89048989 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.327143 9356 raft_consensus.cc:1261] T a0b4daae580844ae828eb73355cbe6d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.327491 10219 consensus_queue.cc:1035] T c3633f6cb6dc4bf3ae9871ae89048989 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.327783 10219 consensus_queue.cc:1035] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.332793 9226 raft_consensus.cc:1261] T a0b4daae580844ae828eb73355cbe6d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.333180 10219 consensus_queue.cc:1035] T a0b4daae580844ae828eb73355cbe6d0 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.334982 9620 raft_consensus.cc:1261] T cfdecd04308f42659df60c13bf56cb7f P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.335343 10234 consensus_queue.cc:1035] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.337038 9488 raft_consensus.cc:1261] T cfdecd04308f42659df60c13bf56cb7f P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.337366 10234 consensus_queue.cc:1035] T cfdecd04308f42659df60c13bf56cb7f P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.784 ms
time per row: 0.0598921 ms
I20240501 19:58:35.399295 9078 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:35.404850 9600 tablet_service.cc:1467] Processing CreateTablet for tablet b99833090f7f4b5ebf3fb4a5ef6fd02c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.405148 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99833090f7f4b5ebf3fb4a5ef6fd02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.405215 9205 tablet_service.cc:1467] Processing CreateTablet for tablet b99833090f7f4b5ebf3fb4a5ef6fd02c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.405478 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99833090f7f4b5ebf3fb4a5ef6fd02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.406934 10369 tablet_bootstrap.cc:492] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:35.407780 9599 tablet_service.cc:1467] Processing CreateTablet for tablet dcd48568731c4409916d7a4a318ffd39 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.407990 10369 tablet_bootstrap.cc:654] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.408035 9599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd48568731c4409916d7a4a318ffd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.408069 9204 tablet_service.cc:1467] Processing CreateTablet for tablet 2d27782af6124800a48b90b9a72dd5cf (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.408282 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d27782af6124800a48b90b9a72dd5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.409957 10369 tablet_bootstrap.cc:492] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:35.410068 10371 tablet_bootstrap.cc:492] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:35.410089 10369 ts_tablet_manager.cc:1367] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:58:35.410181 9206 tablet_service.cc:1467] Processing CreateTablet for tablet dcd48568731c4409916d7a4a318ffd39 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.410384 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd48568731c4409916d7a4a318ffd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.410867 10369 raft_consensus.cc:348] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.411015 10371 tablet_bootstrap.cc:654] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.411213 10369 raft_consensus.cc:374] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.411299 10369 raft_consensus.cc:729] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.411525 10369 consensus_queue.cc:260] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.411891 9335 tablet_service.cc:1467] Processing CreateTablet for tablet b99833090f7f4b5ebf3fb4a5ef6fd02c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.411947 9336 tablet_service.cc:1467] Processing CreateTablet for tablet dcd48568731c4409916d7a4a318ffd39 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.412181 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99833090f7f4b5ebf3fb4a5ef6fd02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.412488 9466 tablet_service.cc:1467] Processing CreateTablet for tablet 2d27782af6124800a48b90b9a72dd5cf (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.412763 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d27782af6124800a48b90b9a72dd5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.414297 9600 tablet_service.cc:1467] Processing CreateTablet for tablet 2d27782af6124800a48b90b9a72dd5cf (DEFAULT_TABLE table=rebalance_test_table_2 [id=d2db04da75674751b503e6d86ceb9104]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:35.414466 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd48568731c4409916d7a4a318ffd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.414502 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d27782af6124800a48b90b9a72dd5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:35.415940 10369 ts_tablet_manager.cc:1398] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:58:35.416111 10369 tablet_bootstrap.cc:492] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:35.417286 10371 tablet_bootstrap.cc:492] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:35.417321 10369 tablet_bootstrap.cc:654] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.417364 10371 ts_tablet_manager.cc:1367] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:58:35.418026 10371 raft_consensus.cc:348] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.418291 10369 tablet_bootstrap.cc:492] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:35.418326 10371 raft_consensus.cc:374] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.418360 10369 ts_tablet_manager.cc:1367] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:35.418366 10371 raft_consensus.cc:729] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.418498 10373 tablet_bootstrap.cc:492] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:35.418507 10371 consensus_queue.cc:260] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.418776 10369 raft_consensus.cc:348] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.418877 10369 raft_consensus.cc:374] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.418910 10369 raft_consensus.cc:729] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.419018 10369 consensus_queue.cc:260] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.419270 10373 tablet_bootstrap.cc:654] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.419283 10369 ts_tablet_manager.cc:1398] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:35.419365 10371 ts_tablet_manager.cc:1398] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:35.419425 10369 tablet_bootstrap.cc:492] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:35.419515 10371 tablet_bootstrap.cc:492] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:35.420213 10373 tablet_bootstrap.cc:492] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:35.420290 10373 ts_tablet_manager.cc:1367] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:35.420315 10369 tablet_bootstrap.cc:654] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.420753 10373 raft_consensus.cc:348] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.420858 10373 raft_consensus.cc:374] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.420888 10373 raft_consensus.cc:729] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.420998 10373 consensus_queue.cc:260] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.421244 10371 tablet_bootstrap.cc:654] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.421272 10373 ts_tablet_manager.cc:1398] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:35.421967 10369 tablet_bootstrap.cc:492] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:35.422045 10369 ts_tablet_manager.cc:1367] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:58:35.422107 10374 tablet_bootstrap.cc:492] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:35.422472 10369 raft_consensus.cc:348] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.422576 10369 raft_consensus.cc:374] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.422619 10369 raft_consensus.cc:729] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.422736 10369 consensus_queue.cc:260] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.422967 10369 ts_tablet_manager.cc:1398] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:35.423120 10374 tablet_bootstrap.cc:654] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.424382 10371 tablet_bootstrap.cc:492] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:35.424422 10374 tablet_bootstrap.cc:492] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:35.424474 10371 ts_tablet_manager.cc:1367] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:58:35.424486 10374 ts_tablet_manager.cc:1367] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:58:35.424880 10371 raft_consensus.cc:348] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.424974 10371 raft_consensus.cc:374] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.424933 10374 raft_consensus.cc:348] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.425004 10371 raft_consensus.cc:729] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.425022 10374 raft_consensus.cc:374] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.425055 10374 raft_consensus.cc:729] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.425125 10371 consensus_queue.cc:260] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.425369 10371 ts_tablet_manager.cc:1398] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:35.425591 10371 tablet_bootstrap.cc:492] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:35.425206 10374 consensus_queue.cc:260] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.426074 10374 ts_tablet_manager.cc:1398] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:58:35.426410 10371 tablet_bootstrap.cc:654] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.427345 10371 tablet_bootstrap.cc:492] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:35.427425 10371 ts_tablet_manager.cc:1367] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:35.427491 10374 tablet_bootstrap.cc:492] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:35.427798 10371 raft_consensus.cc:348] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.427875 10371 raft_consensus.cc:374] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.427910 10371 raft_consensus.cc:729] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.428023 10371 consensus_queue.cc:260] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.428268 10371 ts_tablet_manager.cc:1398] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:35.429527 10374 tablet_bootstrap.cc:654] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:35.430510 10374 tablet_bootstrap.cc:492] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:35.430577 10374 ts_tablet_manager.cc:1367] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:58:35.430883 10374 raft_consensus.cc:348] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.430959 10374 raft_consensus.cc:374] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:35.430984 10374 raft_consensus.cc:729] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:35.431113 10374 consensus_queue.cc:260] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.431331 10374 ts_tablet_manager.cc:1398] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:35.503515 10234 raft_consensus.cc:482] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:35.503625 10234 raft_consensus.cc:504] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.503979 10234 leader_election.cc:288] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:35.504331 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99833090f7f4b5ebf3fb4a5ef6fd02c" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:35.504370 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99833090f7f4b5ebf3fb4a5ef6fd02c" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:35.504457 9226 raft_consensus.cc:2454] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:35.504495 9619 raft_consensus.cc:2454] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:35.504774 9291 leader_election.cc:302] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:35.504930 10234 raft_consensus.cc:2781] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:35.504979 10234 raft_consensus.cc:482] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:35.505031 10234 raft_consensus.cc:3037] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.505849 10234 raft_consensus.cc:504] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.506119 10234 leader_election.cc:288] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:35.506392 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99833090f7f4b5ebf3fb4a5ef6fd02c" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:35.506440 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99833090f7f4b5ebf3fb4a5ef6fd02c" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:35.506505 9226 raft_consensus.cc:3037] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.506567 9619 raft_consensus.cc:3037] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.507251 9226 raft_consensus.cc:2454] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:35.507380 9619 raft_consensus.cc:2454] T b99833090f7f4b5ebf3fb4a5ef6fd02c P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:35.507515 9291 leader_election.cc:302] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:35.507686 10234 raft_consensus.cc:2781] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:35.507772 10234 raft_consensus.cc:686] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:35.507916 10234 consensus_queue.cc:237] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.509248 9077 catalog_manager.cc:5484] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } }
I20240501 19:58:35.571918 10234 raft_consensus.cc:482] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:35.572010 10234 raft_consensus.cc:504] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.572336 10234 leader_election.cc:288] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:35.572679 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd48568731c4409916d7a4a318ffd39" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:35.572714 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd48568731c4409916d7a4a318ffd39" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:35.572791 9226 raft_consensus.cc:2454] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:35.572840 9619 raft_consensus.cc:2454] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:35.573068 9291 leader_election.cc:302] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:35.573194 10234 raft_consensus.cc:2781] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:35.573242 10234 raft_consensus.cc:482] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:35.573266 10234 raft_consensus.cc:3037] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.573956 10234 raft_consensus.cc:504] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.574210 10234 leader_election.cc:288] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:35.574476 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd48568731c4409916d7a4a318ffd39" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:35.574530 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd48568731c4409916d7a4a318ffd39" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:35.574561 9226 raft_consensus.cc:3037] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.574654 9619 raft_consensus.cc:3037] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.575331 9226 raft_consensus.cc:2454] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:35.575479 9619 raft_consensus.cc:2454] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:35.575588 9291 leader_election.cc:302] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:35.575744 10234 raft_consensus.cc:2781] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:35.575819 10234 raft_consensus.cc:686] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:35.575961 10234 consensus_queue.cc:237] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:35.577276 9077 catalog_manager.cc:5484] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } }
I20240501 19:58:35.702553 10221 raft_consensus.cc:482] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:35.702684 10221 raft_consensus.cc:504] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.703744 10221 leader_election.cc:288] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:35.708004 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d27782af6124800a48b90b9a72dd5cf" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:35.708141 9619 raft_consensus.cc:2454] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:35.708333 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d27782af6124800a48b90b9a72dd5cf" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:35.708456 9488 raft_consensus.cc:2454] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:35.708804 9162 leader_election.cc:302] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:35.709023 10221 raft_consensus.cc:2781] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:35.709081 10221 raft_consensus.cc:482] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:35.709117 10221 raft_consensus.cc:3037] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.709924 10221 raft_consensus.cc:504] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.710225 10221 leader_election.cc:288] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:35.710503 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d27782af6124800a48b90b9a72dd5cf" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:35.710521 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d27782af6124800a48b90b9a72dd5cf" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:35.710615 9488 raft_consensus.cc:3037] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.710641 9619 raft_consensus.cc:3037] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:35.711508 9488 raft_consensus.cc:2454] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:35.711508 9619 raft_consensus.cc:2454] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:35.711768 9162 leader_election.cc:302] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:35.711990 10221 raft_consensus.cc:2781] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:35.712198 10221 raft_consensus.cc:686] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 LEADER]: Becoming Leader. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Running, Role: LEADER
I20240501 19:58:35.712479 10221 consensus_queue.cc:237] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:35.714777 9077 catalog_manager.cc:5484] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff reported cstate change: term changed from 0 to 1, leader changed from <none> to 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1). New cstate: current_term: 1 leader_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
W20240501 19:58:35.876643 10389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:35.876856 10389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:35.879981 10389 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:58:35.912600 10234 consensus_queue.cc:1035] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.931370 10288 consensus_queue.cc:1035] T b99833090f7f4b5ebf3fb4a5ef6fd02c P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.961597 9619 raft_consensus.cc:1261] T dcd48568731c4409916d7a4a318ffd39 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.962020 10234 consensus_queue.cc:1035] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.963340 9226 raft_consensus.cc:1261] T dcd48568731c4409916d7a4a318ffd39 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.965305 10288 consensus_queue.cc:1035] T dcd48568731c4409916d7a4a318ffd39 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.967650 9618 raft_consensus.cc:1261] T 2d27782af6124800a48b90b9a72dd5cf P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 37fa3f77f5a644d181ccb7ea7ea15fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.968616 9488 raft_consensus.cc:1261] T 2d27782af6124800a48b90b9a72dd5cf P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 37fa3f77f5a644d181ccb7ea7ea15fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:35.969125 10221 consensus_queue.cc:1035] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:35.969847 10221 consensus_queue.cc:1035] T 2d27782af6124800a48b90b9a72dd5cf P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 100.849 ms
time per row: 0.0504246 ms
I20240501 19:58:36.020062 9074 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:36.025027 9335 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc5c5ff88404f1b95e5fd75dc40662a (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.025326 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc5c5ff88404f1b95e5fd75dc40662a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.025377 9466 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc5c5ff88404f1b95e5fd75dc40662a (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.025548 9205 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc5c5ff88404f1b95e5fd75dc40662a (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.025656 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc5c5ff88404f1b95e5fd75dc40662a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.025795 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc5c5ff88404f1b95e5fd75dc40662a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.027390 10447 tablet_bootstrap.cc:492] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:36.027971 10449 tablet_bootstrap.cc:492] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:36.028079 9467 tablet_service.cc:1467] Processing CreateTablet for tablet 8c349b9237674d31b50ac205d4b12e76 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.028156 9600 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd06dd8c1d483e8be8068545cf9ac5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.028260 10447 tablet_bootstrap.cc:654] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.028306 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c349b9237674d31b50ac205d4b12e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.028406 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd06dd8c1d483e8be8068545cf9ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.028774 10449 tablet_bootstrap.cc:654] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.029868 10449 tablet_bootstrap.cc:492] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:36.029940 10449 ts_tablet_manager.cc:1367] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:36.030437 9335 tablet_service.cc:1467] Processing CreateTablet for tablet 8c349b9237674d31b50ac205d4b12e76 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.030542 10452 tablet_bootstrap.cc:492] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:36.030689 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c349b9237674d31b50ac205d4b12e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.031299 9466 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd06dd8c1d483e8be8068545cf9ac5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.031342 9600 tablet_service.cc:1467] Processing CreateTablet for tablet 8c349b9237674d31b50ac205d4b12e76 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.031445 10452 tablet_bootstrap.cc:654] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.031529 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd06dd8c1d483e8be8068545cf9ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.031565 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c349b9237674d31b50ac205d4b12e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.032689 10449 raft_consensus.cc:348] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.032788 10449 raft_consensus.cc:374] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.032817 10449 raft_consensus.cc:729] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.032963 10449 consensus_queue.cc:260] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.033375 10449 ts_tablet_manager.cc:1398] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:58:36.033587 9205 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd06dd8c1d483e8be8068545cf9ac5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fe0134a9a85f4381bd47c84de6d70155]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.033771 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd06dd8c1d483e8be8068545cf9ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.034420 10448 tablet_bootstrap.cc:492] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:36.035364 10449 tablet_bootstrap.cc:492] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:36.035713 10448 tablet_bootstrap.cc:654] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.036098 10449 tablet_bootstrap.cc:654] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.036124 10447 tablet_bootstrap.cc:492] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:36.036208 10447 ts_tablet_manager.cc:1367] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:58:36.036594 10447 raft_consensus.cc:348] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.036708 10447 raft_consensus.cc:374] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.036748 10447 raft_consensus.cc:729] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.036885 10448 tablet_bootstrap.cc:492] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:36.036881 10447 consensus_queue.cc:260] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.036965 10448 ts_tablet_manager.cc:1367] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:58:36.037109 10449 tablet_bootstrap.cc:492] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:36.037173 10447 ts_tablet_manager.cc:1398] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.037184 10449 ts_tablet_manager.cc:1367] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:36.037334 10447 tablet_bootstrap.cc:492] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:36.037402 10448 raft_consensus.cc:348] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.037495 10448 raft_consensus.cc:374] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.037523 10448 raft_consensus.cc:729] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.037566 10449 raft_consensus.cc:348] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.037669 10449 raft_consensus.cc:374] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.037667 10448 consensus_queue.cc:260] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.037711 10449 raft_consensus.cc:729] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.037851 10449 consensus_queue.cc:260] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.037982 10448 ts_tablet_manager.cc:1398] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.038084 10447 tablet_bootstrap.cc:654] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.038130 10448 tablet_bootstrap.cc:492] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:36.038179 10449 ts_tablet_manager.cc:1398] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.038916 10448 tablet_bootstrap.cc:654] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.040035 10452 tablet_bootstrap.cc:492] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:36.040143 10452 ts_tablet_manager.cc:1367] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:58:36.040647 10452 raft_consensus.cc:348] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.040752 10452 raft_consensus.cc:374] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.040797 10452 raft_consensus.cc:729] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.040889 10448 tablet_bootstrap.cc:492] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:36.040963 10448 ts_tablet_manager.cc:1367] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:58:36.040922 10452 consensus_queue.cc:260] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.041210 10452 ts_tablet_manager.cc:1398] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:36.041407 10452 tablet_bootstrap.cc:492] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:36.041370 10448 raft_consensus.cc:348] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.041457 10448 raft_consensus.cc:374] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.041496 10448 raft_consensus.cc:729] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.041602 10448 consensus_queue.cc:260] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.041994 10448 ts_tablet_manager.cc:1398] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:36.042276 10452 tablet_bootstrap.cc:654] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.043323 10452 tablet_bootstrap.cc:492] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:36.043412 10452 ts_tablet_manager.cc:1367] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:36.043440 10447 tablet_bootstrap.cc:492] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:36.043520 10447 ts_tablet_manager.cc:1367] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:58:36.043803 10452 raft_consensus.cc:348] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.043917 10452 raft_consensus.cc:374] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.043876 10447 raft_consensus.cc:348] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.043954 10452 raft_consensus.cc:729] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.043964 10447 raft_consensus.cc:374] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.043998 10447 raft_consensus.cc:729] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.044106 10452 consensus_queue.cc:260] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.044108 10447 consensus_queue.cc:260] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.044364 10447 ts_tablet_manager.cc:1398] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.044430 10452 ts_tablet_manager.cc:1398] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.044515 10447 tablet_bootstrap.cc:492] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:36.045153 10447 tablet_bootstrap.cc:654] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.046152 10447 tablet_bootstrap.cc:492] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:36.046257 10447 ts_tablet_manager.cc:1367] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:36.047215 10447 raft_consensus.cc:348] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.047297 10447 raft_consensus.cc:374] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.047330 10447 raft_consensus.cc:729] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.047447 10447 consensus_queue.cc:260] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.047698 10447 ts_tablet_manager.cc:1398] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.068271 10221 raft_consensus.cc:482] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:36.068358 10221 raft_consensus.cc:504] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.068722 10221 leader_election.cc:288] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:36.069080 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc5c5ff88404f1b95e5fd75dc40662a" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:36.069206 9488 raft_consensus.cc:2454] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:36.069478 9162 leader_election.cc:302] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:36.069653 10221 raft_consensus.cc:2781] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:36.069705 10221 raft_consensus.cc:482] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:36.069742 10221 raft_consensus.cc:3037] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.070494 10221 raft_consensus.cc:504] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.070828 10221 leader_election.cc:288] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Requested vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:36.071120 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc5c5ff88404f1b95e5fd75dc40662a" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:36.071223 9488 raft_consensus.cc:3037] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.071995 9488 raft_consensus.cc:2454] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:36.072121 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc5c5ff88404f1b95e5fd75dc40662a" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:36.072140 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc5c5ff88404f1b95e5fd75dc40662a" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:36.072264 9357 raft_consensus.cc:2454] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:36.072260 9162 leader_election.cc:302] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:36.072423 10221 raft_consensus.cc:2781] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:36.072536 10221 raft_consensus.cc:686] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 LEADER]: Becoming Leader. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Running, Role: LEADER
I20240501 19:58:36.072685 10221 consensus_queue.cc:237] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.073719 10219 raft_consensus.cc:482] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:36.073792 10219 raft_consensus.cc:504] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.074084 10219 leader_election.cc:288] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:36.074162 9082 catalog_manager.cc:5484] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff reported cstate change: term changed from 0 to 1, leader changed from <none> to 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1). New cstate: current_term: 1 leader_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } }
I20240501 19:58:36.074456 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd06dd8c1d483e8be8068545cf9ac5" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:36.074540 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd06dd8c1d483e8be8068545cf9ac5" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:36.074564 9224 raft_consensus.cc:2454] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:36.074668 9620 raft_consensus.cc:2454] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:36.075336 9422 leader_election.cc:302] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:36.075505 10219 raft_consensus.cc:2781] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:36.075565 10219 raft_consensus.cc:482] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:36.075613 10219 raft_consensus.cc:3037] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.076400 10219 raft_consensus.cc:504] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.076653 10219 leader_election.cc:288] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:36.076943 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd06dd8c1d483e8be8068545cf9ac5" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:36.076973 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd06dd8c1d483e8be8068545cf9ac5" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:36.077056 9224 raft_consensus.cc:3037] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.077056 9620 raft_consensus.cc:3037] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.078051 9620 raft_consensus.cc:2454] T d4cd06dd8c1d483e8be8068545cf9ac5 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:36.078219 9224 raft_consensus.cc:2454] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:36.078320 9422 leader_election.cc:302] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:36.078483 10219 raft_consensus.cc:2781] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:36.078661 10219 raft_consensus.cc:686] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:36.078776 10219 consensus_queue.cc:237] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.080005 9082 catalog_manager.cc:5484] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } }
I20240501 19:58:36.316047 10219 raft_consensus.cc:482] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:36.316164 10219 raft_consensus.cc:504] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.316534 10219 leader_election.cc:288] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:36.316869 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c349b9237674d31b50ac205d4b12e76" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:36.316975 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c349b9237674d31b50ac205d4b12e76" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:36.317023 9357 raft_consensus.cc:2454] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:36.317119 9620 raft_consensus.cc:2454] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:36.317299 9421 leader_election.cc:302] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:36.317489 10219 raft_consensus.cc:2781] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:36.317554 10219 raft_consensus.cc:482] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:36.317811 10219 raft_consensus.cc:3037] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.318578 10219 raft_consensus.cc:504] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.318874 10219 leader_election.cc:288] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:36.319149 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c349b9237674d31b50ac205d4b12e76" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:36.319203 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c349b9237674d31b50ac205d4b12e76" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:36.319262 9357 raft_consensus.cc:3037] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.319320 9620 raft_consensus.cc:3037] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.320202 9357 raft_consensus.cc:2454] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:36.320202 9620 raft_consensus.cc:2454] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:36.320463 9422 leader_election.cc:302] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:36.320617 10219 raft_consensus.cc:2781] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:36.320708 10219 raft_consensus.cc:686] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:36.320835 10219 consensus_queue.cc:237] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:36.322321 9082 catalog_manager.cc:5484] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } }
W20240501 19:58:36.495151 10464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:36.495419 10464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:36.498291 10464 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:58:36.578755 10219 consensus_queue.cc:1035] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:36.580708 9224 raft_consensus.cc:1261] T d4cd06dd8c1d483e8be8068545cf9ac5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:36.581108 10219 consensus_queue.cc:1035] T d4cd06dd8c1d483e8be8068545cf9ac5 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:36.581374 9357 raft_consensus.cc:1261] T 8c349b9237674d31b50ac205d4b12e76 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:36.581663 9488 raft_consensus.cc:1261] T 2bc5c5ff88404f1b95e5fd75dc40662a P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 37fa3f77f5a644d181ccb7ea7ea15fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:36.581916 10219 consensus_queue.cc:1035] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:36.582283 9619 raft_consensus.cc:1261] T 8c349b9237674d31b50ac205d4b12e76 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:36.582788 10219 consensus_queue.cc:1035] T 8c349b9237674d31b50ac205d4b12e76 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:36.583344 10221 consensus_queue.cc:1035] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:36.585999 9356 raft_consensus.cc:3037] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.587241 9356 raft_consensus.cc:1261] T 2bc5c5ff88404f1b95e5fd75dc40662a P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer 37fa3f77f5a644d181ccb7ea7ea15fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:36.587759 10221 consensus_queue.cc:1035] T 2bc5c5ff88404f1b95e5fd75dc40662a P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.808 ms
time per row: 0.0519041 ms
I20240501 19:58:36.640913 9082 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:36.645838 9205 tablet_service.cc:1467] Processing CreateTablet for tablet c3cdab156e5f4039ae092eb8027ecb56 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.646054 9466 tablet_service.cc:1467] Processing CreateTablet for tablet c3cdab156e5f4039ae092eb8027ecb56 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.646152 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3cdab156e5f4039ae092eb8027ecb56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.646185 9600 tablet_service.cc:1467] Processing CreateTablet for tablet 17a998e2707a4fdca25ec6f7f3898ba8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.646309 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3cdab156e5f4039ae092eb8027ecb56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.646450 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a998e2707a4fdca25ec6f7f3898ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.646562 9335 tablet_service.cc:1467] Processing CreateTablet for tablet c3cdab156e5f4039ae092eb8027ecb56 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.646812 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3cdab156e5f4039ae092eb8027ecb56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.647460 9206 tablet_service.cc:1467] Processing CreateTablet for tablet 17a998e2707a4fdca25ec6f7f3898ba8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.647684 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a998e2707a4fdca25ec6f7f3898ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.649081 9336 tablet_service.cc:1467] Processing CreateTablet for tablet 17a998e2707a4fdca25ec6f7f3898ba8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.649186 9599 tablet_service.cc:1467] Processing CreateTablet for tablet 67d40d8cb33b4aa5857b4bdcf6c0c0a5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.649299 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a998e2707a4fdca25ec6f7f3898ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.649394 9599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d40d8cb33b4aa5857b4bdcf6c0c0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.649513 10527 tablet_bootstrap.cc:492] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:36.650341 10527 tablet_bootstrap.cc:654] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.650887 10530 tablet_bootstrap.cc:492] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:36.651286 9206 tablet_service.cc:1467] Processing CreateTablet for tablet 67d40d8cb33b4aa5857b4bdcf6c0c0a5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.651381 10527 tablet_bootstrap.cc:492] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:36.651468 10527 ts_tablet_manager.cc:1367] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:36.651484 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d40d8cb33b4aa5857b4bdcf6c0c0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.651734 10530 tablet_bootstrap.cc:654] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.651870 10527 raft_consensus.cc:348] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.651988 10527 raft_consensus.cc:374] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.652031 10527 raft_consensus.cc:729] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.652153 10527 consensus_queue.cc:260] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.652439 10527 ts_tablet_manager.cc:1398] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.652400 9466 tablet_service.cc:1467] Processing CreateTablet for tablet 67d40d8cb33b4aa5857b4bdcf6c0c0a5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=be243a51e9d442609bf4ea41b43d5620]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:36.652606 10527 tablet_bootstrap.cc:492] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:36.652626 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d40d8cb33b4aa5857b4bdcf6c0c0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:36.652947 10530 tablet_bootstrap.cc:492] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:36.653026 10530 ts_tablet_manager.cc:1367] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:36.653434 10527 tablet_bootstrap.cc:654] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.653578 10530 raft_consensus.cc:348] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.653672 10530 raft_consensus.cc:374] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.653703 10530 raft_consensus.cc:729] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.653823 10530 consensus_queue.cc:260] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.654075 10530 ts_tablet_manager.cc:1398] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.654914 10528 tablet_bootstrap.cc:492] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:36.655723 10529 tablet_bootstrap.cc:492] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:36.655795 10528 tablet_bootstrap.cc:654] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.655889 10527 tablet_bootstrap.cc:492] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:36.655968 10527 ts_tablet_manager.cc:1367] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:58:36.656069 10530 tablet_bootstrap.cc:492] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:36.656373 10527 raft_consensus.cc:348] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.656469 10527 raft_consensus.cc:374] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.656502 10527 raft_consensus.cc:729] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.656605 10527 consensus_queue.cc:260] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.656841 10529 tablet_bootstrap.cc:654] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.656895 10527 ts_tablet_manager.cc:1398] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.656944 10530 tablet_bootstrap.cc:654] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.657040 10528 tablet_bootstrap.cc:492] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:36.657052 10527 tablet_bootstrap.cc:492] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:36.657119 10528 ts_tablet_manager.cc:1367] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:36.657559 10528 raft_consensus.cc:348] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.657665 10528 raft_consensus.cc:374] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.657699 10528 raft_consensus.cc:729] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.657809 10528 consensus_queue.cc:260] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.658052 10528 ts_tablet_manager.cc:1398] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.658115 10527 tablet_bootstrap.cc:654] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.658210 10528 tablet_bootstrap.cc:492] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:36.659099 10528 tablet_bootstrap.cc:654] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.659739 10530 tablet_bootstrap.cc:492] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:36.659812 10530 ts_tablet_manager.cc:1367] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:58:36.660115 10529 tablet_bootstrap.cc:492] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:36.660195 10529 ts_tablet_manager.cc:1367] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:58:36.660316 10528 tablet_bootstrap.cc:492] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:36.660244 10530 raft_consensus.cc:348] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.660367 10530 raft_consensus.cc:374] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.660382 10528 ts_tablet_manager.cc:1367] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:58:36.660410 10530 raft_consensus.cc:729] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.660530 10530 consensus_queue.cc:260] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.660662 10529 raft_consensus.cc:348] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.660761 10529 raft_consensus.cc:374] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.660789 10530 ts_tablet_manager.cc:1398] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.660801 10529 raft_consensus.cc:729] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.660910 10529 consensus_queue.cc:260] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.661011 10528 raft_consensus.cc:348] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.661093 10528 raft_consensus.cc:374] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.661123 10528 raft_consensus.cc:729] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.661202 10529 ts_tablet_manager.cc:1398] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.661221 10528 consensus_queue.cc:260] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.661360 10529 tablet_bootstrap.cc:492] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:36.661444 10528 ts_tablet_manager.cc:1398] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:36.662156 10529 tablet_bootstrap.cc:654] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:36.663220 10527 tablet_bootstrap.cc:492] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:36.663268 10529 tablet_bootstrap.cc:492] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:36.663332 10527 ts_tablet_manager.cc:1367] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:58:36.663344 10529 ts_tablet_manager.cc:1367] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:36.663707 10527 raft_consensus.cc:348] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.663748 10529 raft_consensus.cc:348] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.663798 10527 raft_consensus.cc:374] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.663831 10527 raft_consensus.cc:729] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.663834 10529 raft_consensus.cc:374] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:36.663866 10529 raft_consensus.cc:729] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:36.663944 10527 consensus_queue.cc:260] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.663965 10529 consensus_queue.cc:260] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.664216 10529 ts_tablet_manager.cc:1398] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:36.664321 10527 ts_tablet_manager.cc:1398] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:36.842988 10426 raft_consensus.cc:482] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:36.843142 10426 raft_consensus.cc:504] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.843523 10426 leader_election.cc:288] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:36.843861 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3cdab156e5f4039ae092eb8027ecb56" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:36.843884 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3cdab156e5f4039ae092eb8027ecb56" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:36.843986 9224 raft_consensus.cc:2454] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:36.844004 9488 raft_consensus.cc:2454] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:36.844272 9293 leader_election.cc:302] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:36.844413 10426 raft_consensus.cc:2781] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:36.844470 10426 raft_consensus.cc:482] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:36.844504 10426 raft_consensus.cc:3037] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.845266 10426 raft_consensus.cc:504] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.845592 10426 leader_election.cc:288] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:36.845891 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3cdab156e5f4039ae092eb8027ecb56" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:36.845917 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3cdab156e5f4039ae092eb8027ecb56" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:36.846000 9488 raft_consensus.cc:3037] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.846016 9224 raft_consensus.cc:3037] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.846940 9224 raft_consensus.cc:2454] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:36.846940 9488 raft_consensus.cc:2454] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:36.847227 9293 leader_election.cc:302] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:36.847396 10426 raft_consensus.cc:2781] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:36.847481 10426 raft_consensus.cc:686] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:36.847615 10426 consensus_queue.cc:237] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:36.849079 9082 catalog_manager.cc:5484] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
I20240501 19:58:36.900777 10426 raft_consensus.cc:482] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:36.900885 10426 raft_consensus.cc:504] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.901154 10426 leader_election.cc:288] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:36.901494 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a998e2707a4fdca25ec6f7f3898ba8" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:36.901463 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a998e2707a4fdca25ec6f7f3898ba8" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:36.901612 9620 raft_consensus.cc:2454] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:36.901612 9224 raft_consensus.cc:2454] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:36.901850 9291 leader_election.cc:302] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:36.902024 10426 raft_consensus.cc:2781] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:36.902086 10426 raft_consensus.cc:482] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:36.902132 10426 raft_consensus.cc:3037] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.902835 10426 raft_consensus.cc:504] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.903151 10426 leader_election.cc:288] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:36.903419 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a998e2707a4fdca25ec6f7f3898ba8" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:36.903471 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a998e2707a4fdca25ec6f7f3898ba8" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:36.903533 9224 raft_consensus.cc:3037] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.903563 9620 raft_consensus.cc:3037] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.904214 9620 raft_consensus.cc:2454] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:36.904223 9224 raft_consensus.cc:2454] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:36.904444 9291 leader_election.cc:302] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:36.904589 10426 raft_consensus.cc:2781] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:36.904675 10426 raft_consensus.cc:686] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:36.904845 10426 consensus_queue.cc:237] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:36.906253 9082 catalog_manager.cc:5484] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } }
I20240501 19:58:36.913761 10219 raft_consensus.cc:482] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:36.913848 10219 raft_consensus.cc:504] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.914119 10219 leader_election.cc:288] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:36.914374 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d40d8cb33b4aa5857b4bdcf6c0c0a5" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:36.914435 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d40d8cb33b4aa5857b4bdcf6c0c0a5" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:36.914481 9620 raft_consensus.cc:2454] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:36.914526 9224 raft_consensus.cc:2454] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:36.914700 9422 leader_election.cc:302] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:36.914837 10219 raft_consensus.cc:2781] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:36.914886 10219 raft_consensus.cc:482] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:36.914932 10219 raft_consensus.cc:3037] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.915619 10219 raft_consensus.cc:504] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.915889 10219 leader_election.cc:288] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:36.916146 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d40d8cb33b4aa5857b4bdcf6c0c0a5" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:36.916236 9620 raft_consensus.cc:3037] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.916224 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d40d8cb33b4aa5857b4bdcf6c0c0a5" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:36.916308 9224 raft_consensus.cc:3037] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:36.917039 9620 raft_consensus.cc:2454] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:36.917045 9224 raft_consensus.cc:2454] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:36.917273 9422 leader_election.cc:302] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:36.917430 10219 raft_consensus.cc:2781] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:36.917511 10219 raft_consensus.cc:686] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:36.917629 10219 consensus_queue.cc:237] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:36.918920 9082 catalog_manager.cc:5484] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } }
W20240501 19:58:37.038367 10543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:37.038604 10543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:37.041544 10543 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:58:37.108997 9620 raft_consensus.cc:1261] T 17a998e2707a4fdca25ec6f7f3898ba8 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.109335 9224 raft_consensus.cc:1261] T 17a998e2707a4fdca25ec6f7f3898ba8 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.109649 10426 consensus_queue.cc:1035] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.110949 9488 raft_consensus.cc:1261] T c3cdab156e5f4039ae092eb8027ecb56 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.111415 10234 consensus_queue.cc:1035] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.114261 9224 raft_consensus.cc:1261] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.114759 9619 raft_consensus.cc:1261] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.115259 10426 consensus_queue.cc:1035] T 17a998e2707a4fdca25ec6f7f3898ba8 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.115612 9224 raft_consensus.cc:1261] T c3cdab156e5f4039ae092eb8027ecb56 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.118583 10461 consensus_queue.cc:1035] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.118944 10461 consensus_queue.cc:1035] T 67d40d8cb33b4aa5857b4bdcf6c0c0a5 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.119086 10234 consensus_queue.cc:1035] T c3cdab156e5f4039ae092eb8027ecb56 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 82.1036 ms
time per row: 0.0410518 ms
I20240501 19:58:37.166436 9082 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:37.174342 9336 tablet_service.cc:1467] Processing CreateTablet for tablet 4e26dcd5531445a69d0796af8ae3aa8c (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.174660 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e26dcd5531445a69d0796af8ae3aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.175310 9206 tablet_service.cc:1467] Processing CreateTablet for tablet 8714523c8f8a41c295259b4df4e979b3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.175612 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8714523c8f8a41c295259b4df4e979b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.175835 9468 tablet_service.cc:1467] Processing CreateTablet for tablet 8714523c8f8a41c295259b4df4e979b3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.176108 9468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8714523c8f8a41c295259b4df4e979b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.176079 9466 tablet_service.cc:1467] Processing CreateTablet for tablet 4e26dcd5531445a69d0796af8ae3aa8c (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.176307 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e26dcd5531445a69d0796af8ae3aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.177192 9600 tablet_service.cc:1467] Processing CreateTablet for tablet 4e26dcd5531445a69d0796af8ae3aa8c (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.177482 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e26dcd5531445a69d0796af8ae3aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.178309 10598 tablet_bootstrap.cc:492] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:37.179164 10598 tablet_bootstrap.cc:654] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.179586 9599 tablet_service.cc:1467] Processing CreateTablet for tablet 362e1f881e1448b68be7df2dea244567 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.179797 9599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362e1f881e1448b68be7df2dea244567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.180322 10597 tablet_bootstrap.cc:492] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:37.181195 10597 tablet_bootstrap.cc:654] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.181948 9467 tablet_service.cc:1467] Processing CreateTablet for tablet 362e1f881e1448b68be7df2dea244567 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.182200 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362e1f881e1448b68be7df2dea244567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.182422 10598 tablet_bootstrap.cc:492] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:37.182489 10598 ts_tablet_manager.cc:1367] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:58:37.182723 10600 tablet_bootstrap.cc:492] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:37.182821 10598 raft_consensus.cc:348] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.182910 10598 raft_consensus.cc:374] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.182935 10598 raft_consensus.cc:729] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.183284 10598 consensus_queue.cc:260] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.183619 10598 ts_tablet_manager.cc:1398] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:37.183863 10598 tablet_bootstrap.cc:492] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:37.185295 9335 tablet_service.cc:1467] Processing CreateTablet for tablet 8714523c8f8a41c295259b4df4e979b3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.185513 10598 tablet_bootstrap.cc:654] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.185537 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8714523c8f8a41c295259b4df4e979b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.186187 10600 tablet_bootstrap.cc:654] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.187112 10597 tablet_bootstrap.cc:492] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:37.187197 10597 ts_tablet_manager.cc:1367] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:58:37.187615 10597 raft_consensus.cc:348] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.187772 10597 raft_consensus.cc:374] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.187815 10597 raft_consensus.cc:729] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.187947 10597 consensus_queue.cc:260] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.188215 10597 ts_tablet_manager.cc:1398] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:37.190016 9336 tablet_service.cc:1467] Processing CreateTablet for tablet 362e1f881e1448b68be7df2dea244567 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cc4d2864cb7148dab0a575f9f75ce0b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.190239 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362e1f881e1448b68be7df2dea244567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.191220 10596 tablet_bootstrap.cc:492] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:37.192126 10596 tablet_bootstrap.cc:654] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.193251 10600 tablet_bootstrap.cc:492] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:37.193356 10600 ts_tablet_manager.cc:1367] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:58:37.193877 10600 raft_consensus.cc:348] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.194008 10600 raft_consensus.cc:374] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.194051 10600 raft_consensus.cc:729] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.194188 10600 consensus_queue.cc:260] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.194485 10600 ts_tablet_manager.cc:1398] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:37.194655 10600 tablet_bootstrap.cc:492] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:37.195762 10600 tablet_bootstrap.cc:654] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.196581 10596 tablet_bootstrap.cc:492] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:37.196707 10596 ts_tablet_manager.cc:1367] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:58:37.196815 10600 tablet_bootstrap.cc:492] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:37.196902 10600 ts_tablet_manager.cc:1367] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:37.197181 10596 raft_consensus.cc:348] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.197253 10600 raft_consensus.cc:348] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.197302 10596 raft_consensus.cc:374] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.197350 10600 raft_consensus.cc:374] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.197391 10600 raft_consensus.cc:729] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.197363 10596 raft_consensus.cc:729] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.197510 10600 consensus_queue.cc:260] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.197564 10596 consensus_queue.cc:260] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.197782 10600 ts_tablet_manager.cc:1398] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:37.197921 10596 ts_tablet_manager.cc:1398] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:37.198077 10596 tablet_bootstrap.cc:492] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:37.198849 10596 tablet_bootstrap.cc:654] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.200081 10598 tablet_bootstrap.cc:492] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:37.200649 10596 tablet_bootstrap.cc:492] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:37.200731 10596 ts_tablet_manager.cc:1367] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:58:37.201133 10596 raft_consensus.cc:348] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.201220 10596 raft_consensus.cc:374] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.201258 10596 raft_consensus.cc:729] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.201377 10596 consensus_queue.cc:260] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.200183 10598 ts_tablet_manager.cc:1367] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.001s
I20240501 19:58:37.201932 10598 raft_consensus.cc:348] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.202039 10598 raft_consensus.cc:374] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.202072 10598 raft_consensus.cc:729] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.202188 10598 consensus_queue.cc:260] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.203018 10596 ts_tablet_manager.cc:1398] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:58:37.203279 10598 ts_tablet_manager.cc:1398] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:58:37.203442 10598 tablet_bootstrap.cc:492] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:37.204432 10598 tablet_bootstrap.cc:654] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.205005 10596 tablet_bootstrap.cc:492] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:37.205694 10596 tablet_bootstrap.cc:654] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.206571 10598 tablet_bootstrap.cc:492] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:37.206656 10598 ts_tablet_manager.cc:1367] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:58:37.207157 10598 raft_consensus.cc:348] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.207275 10598 raft_consensus.cc:374] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.207327 10598 raft_consensus.cc:729] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.207477 10598 consensus_queue.cc:260] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.208007 10596 tablet_bootstrap.cc:492] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:37.208102 10596 ts_tablet_manager.cc:1367] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:58:37.208595 10596 raft_consensus.cc:348] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.208709 10596 raft_consensus.cc:374] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.208756 10596 raft_consensus.cc:729] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.208894 10596 consensus_queue.cc:260] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.209213 10596 ts_tablet_manager.cc:1398] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:37.209651 10598 ts_tablet_manager.cc:1398] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:58:37.244935 10586 raft_consensus.cc:482] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:37.245067 10586 raft_consensus.cc:504] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.245481 10586 leader_election.cc:288] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:37.245915 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8714523c8f8a41c295259b4df4e979b3" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:37.245915 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8714523c8f8a41c295259b4df4e979b3" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:37.246064 9226 raft_consensus.cc:2454] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:37.246064 9488 raft_consensus.cc:2454] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:37.246357 9293 leader_election.cc:302] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:37.246526 10586 raft_consensus.cc:2781] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:37.246587 10586 raft_consensus.cc:482] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:37.246624 10586 raft_consensus.cc:3037] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.247457 10586 raft_consensus.cc:504] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.247761 10586 leader_election.cc:288] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:37.248039 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8714523c8f8a41c295259b4df4e979b3" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:37.248046 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8714523c8f8a41c295259b4df4e979b3" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:37.248145 9488 raft_consensus.cc:3037] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.248162 9226 raft_consensus.cc:3037] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.248880 9488 raft_consensus.cc:2454] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:37.248975 9226 raft_consensus.cc:2454] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:37.249119 9293 leader_election.cc:302] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:37.249269 10586 raft_consensus.cc:2781] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:37.249366 10586 raft_consensus.cc:686] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:37.249506 10586 consensus_queue.cc:237] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.251175 9080 catalog_manager.cc:5484] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
I20240501 19:58:37.314038 10576 raft_consensus.cc:482] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:37.314157 10576 raft_consensus.cc:504] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.314524 10576 leader_election.cc:288] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:37.314874 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e26dcd5531445a69d0796af8ae3aa8c" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:37.314898 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e26dcd5531445a69d0796af8ae3aa8c" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:37.315006 9356 raft_consensus.cc:2454] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:37.315068 9619 raft_consensus.cc:2454] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:37.315326 9421 leader_election.cc:302] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:37.315480 10576 raft_consensus.cc:2781] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:37.315536 10576 raft_consensus.cc:482] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:37.315579 10576 raft_consensus.cc:3037] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.316424 10576 raft_consensus.cc:504] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.316726 10576 leader_election.cc:288] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:37.317015 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e26dcd5531445a69d0796af8ae3aa8c" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:37.317026 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e26dcd5531445a69d0796af8ae3aa8c" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:37.317116 9619 raft_consensus.cc:3037] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.317116 9356 raft_consensus.cc:3037] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.317865 9356 raft_consensus.cc:2454] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:37.317924 9619 raft_consensus.cc:2454] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:37.318176 9422 leader_election.cc:302] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:37.318326 10576 raft_consensus.cc:2781] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:37.318409 10576 raft_consensus.cc:686] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:37.318569 10576 consensus_queue.cc:237] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.320225 9080 catalog_manager.cc:5484] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
I20240501 19:58:37.410131 10586 raft_consensus.cc:482] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:37.410243 10586 raft_consensus.cc:504] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.410621 10586 leader_election.cc:288] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:37.410972 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362e1f881e1448b68be7df2dea244567" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:37.410969 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362e1f881e1448b68be7df2dea244567" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:37.411131 9619 raft_consensus.cc:2454] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:37.411145 9488 raft_consensus.cc:2454] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:37.411402 9291 leader_election.cc:302] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:37.411577 10586 raft_consensus.cc:2781] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:37.411629 10586 raft_consensus.cc:482] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:37.411662 10586 raft_consensus.cc:3037] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.412469 10586 raft_consensus.cc:504] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.412775 10586 leader_election.cc:288] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:37.413095 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362e1f881e1448b68be7df2dea244567" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:37.413087 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362e1f881e1448b68be7df2dea244567" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:37.413190 9488 raft_consensus.cc:3037] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.413189 9619 raft_consensus.cc:3037] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.413928 9619 raft_consensus.cc:2454] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:37.414153 9291 leader_election.cc:302] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:37.414314 10586 raft_consensus.cc:2781] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:37.414409 10586 raft_consensus.cc:686] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:37.414563 10586 consensus_queue.cc:237] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:37.416119 9080 catalog_manager.cc:5484] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } }
I20240501 19:58:37.431802 9488 raft_consensus.cc:2454] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
W20240501 19:58:37.577859 10612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:37.578050 10612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:37.580832 10612 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 19:58:37.662035 9619 raft_consensus.cc:1261] T 362e1f881e1448b68be7df2dea244567 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.662031 9488 raft_consensus.cc:1261] T 362e1f881e1448b68be7df2dea244567 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.662464 10586 consensus_queue.cc:1035] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.662631 10426 consensus_queue.cc:1035] T 362e1f881e1448b68be7df2dea244567 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.663429 9489 raft_consensus.cc:1261] T 8714523c8f8a41c295259b4df4e979b3 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.663589 9226 raft_consensus.cc:1261] T 8714523c8f8a41c295259b4df4e979b3 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.664620 10426 consensus_queue.cc:1035] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.664899 10426 consensus_queue.cc:1035] T 8714523c8f8a41c295259b4df4e979b3 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.667645 9620 raft_consensus.cc:1261] T 4e26dcd5531445a69d0796af8ae3aa8c P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.668762 10576 consensus_queue.cc:1035] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.669588 9356 raft_consensus.cc:1261] T 4e26dcd5531445a69d0796af8ae3aa8c P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:37.670152 10461 consensus_queue.cc:1035] T 4e26dcd5531445a69d0796af8ae3aa8c P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:37.677459 10645 mvcc.cc:205] Tried to move back new op lower bound from 7022975048360529920 to 7022975046938005504. Current Snapshot: MvccSnapshot[applied={T|T < 7022975048360529920}]
I20240501 19:58:37.681865 10644 mvcc.cc:205] Tried to move back new op lower bound from 7022975048360529920 to 7022975046938005504. Current Snapshot: MvccSnapshot[applied={T|T < 7022975048360529920}]
I20240501 19:58:37.689532 10651 mvcc.cc:205] Tried to move back new op lower bound from 7022975048345206784 to 7022975046655217664. Current Snapshot: MvccSnapshot[applied={T|T < 7022975048345206784}]
INSERT report
rows total: 2000
time total: 111.974 ms
time per row: 0.0559868 ms
I20240501 19:58:37.731283 9083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:37.736438 9600 tablet_service.cc:1467] Processing CreateTablet for tablet df84ae9bb9e84293aa9bfe474279bedc (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.736667 9599 tablet_service.cc:1467] Processing CreateTablet for tablet fa70a0afeddd463baef865bf55eadf7f (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.736764 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df84ae9bb9e84293aa9bfe474279bedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.736716 9336 tablet_service.cc:1467] Processing CreateTablet for tablet df84ae9bb9e84293aa9bfe474279bedc (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.736934 9599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa70a0afeddd463baef865bf55eadf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.736963 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df84ae9bb9e84293aa9bfe474279bedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.737702 9467 tablet_service.cc:1467] Processing CreateTablet for tablet fa70a0afeddd463baef865bf55eadf7f (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.737959 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa70a0afeddd463baef865bf55eadf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.738441 9598 tablet_service.cc:1467] Processing CreateTablet for tablet 4d282530a4d442bcbb47f0cbac273f24 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.738670 9598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d282530a4d442bcbb47f0cbac273f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.739276 10672 tablet_bootstrap.cc:492] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:37.740170 10672 tablet_bootstrap.cc:654] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.740219 10674 tablet_bootstrap.cc:492] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:37.740609 9206 tablet_service.cc:1467] Processing CreateTablet for tablet df84ae9bb9e84293aa9bfe474279bedc (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.740887 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df84ae9bb9e84293aa9bfe474279bedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.741025 10674 tablet_bootstrap.cc:654] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.741544 10673 tablet_bootstrap.cc:492] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:37.741660 9205 tablet_service.cc:1467] Processing CreateTablet for tablet fa70a0afeddd463baef865bf55eadf7f (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.741801 9336 tablet_service.cc:1467] Processing CreateTablet for tablet 4d282530a4d442bcbb47f0cbac273f24 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.741854 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa70a0afeddd463baef865bf55eadf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.742029 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d282530a4d442bcbb47f0cbac273f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.742400 10673 tablet_bootstrap.cc:654] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.742493 9204 tablet_service.cc:1467] Processing CreateTablet for tablet 4d282530a4d442bcbb47f0cbac273f24 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cc29c2b436a48b092c9a71bf914c2ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:37.742700 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d282530a4d442bcbb47f0cbac273f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:37.742964 10672 tablet_bootstrap.cc:492] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:37.743072 10672 ts_tablet_manager.cc:1367] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:58:37.743531 10672 raft_consensus.cc:348] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.743640 10672 raft_consensus.cc:374] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.743687 10672 raft_consensus.cc:729] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.743808 10672 consensus_queue.cc:260] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.744084 10672 ts_tablet_manager.cc:1398] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:37.744282 10672 tablet_bootstrap.cc:492] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:37.744678 10674 tablet_bootstrap.cc:492] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:37.744764 10674 ts_tablet_manager.cc:1367] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:58:37.745098 10672 tablet_bootstrap.cc:654] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.745268 10674 raft_consensus.cc:348] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.745384 10674 raft_consensus.cc:374] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.745433 10674 raft_consensus.cc:729] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.745610 10674 consensus_queue.cc:260] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.745981 10674 ts_tablet_manager.cc:1398] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:37.746151 10672 tablet_bootstrap.cc:492] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:37.746361 10672 ts_tablet_manager.cc:1367] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:37.746830 10672 raft_consensus.cc:348] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.746929 10672 raft_consensus.cc:374] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.746959 10672 raft_consensus.cc:729] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.747123 10672 consensus_queue.cc:260] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.747424 10672 ts_tablet_manager.cc:1398] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:37.763496 10673 tablet_bootstrap.cc:492] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:37.763692 10673 ts_tablet_manager.cc:1367] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 19:58:37.764379 10673 raft_consensus.cc:348] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.764570 10673 raft_consensus.cc:374] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.764643 10673 raft_consensus.cc:729] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.764813 10673 consensus_queue.cc:260] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.765324 10679 tablet_bootstrap.cc:492] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:37.765834 10673 ts_tablet_manager.cc:1398] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:37.766037 10673 tablet_bootstrap.cc:492] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:37.766260 10679 tablet_bootstrap.cc:654] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.767081 10673 tablet_bootstrap.cc:654] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.767282 10679 tablet_bootstrap.cc:492] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:37.767366 10679 ts_tablet_manager.cc:1367] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:37.767748 10679 raft_consensus.cc:348] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.767855 10679 raft_consensus.cc:374] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.767886 10679 raft_consensus.cc:729] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.768002 10679 consensus_queue.cc:260] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.768088 10673 tablet_bootstrap.cc:492] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:37.768172 10673 ts_tablet_manager.cc:1367] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:58:37.768262 10679 ts_tablet_manager.cc:1398] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:37.768436 10679 tablet_bootstrap.cc:492] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:37.768564 10673 raft_consensus.cc:348] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.768661 10673 raft_consensus.cc:374] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.768692 10673 raft_consensus.cc:729] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.768796 10673 consensus_queue.cc:260] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.769060 10673 ts_tablet_manager.cc:1398] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:37.769161 10679 tablet_bootstrap.cc:654] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.769307 10673 tablet_bootstrap.cc:492] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:37.770157 10673 tablet_bootstrap.cc:654] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.770673 10679 tablet_bootstrap.cc:492] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:37.770798 10679 ts_tablet_manager.cc:1367] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:58:37.771314 10679 raft_consensus.cc:348] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.771392 10679 raft_consensus.cc:374] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.771420 10679 raft_consensus.cc:729] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.771524 10679 consensus_queue.cc:260] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.771848 10673 tablet_bootstrap.cc:492] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:37.771849 10679 ts_tablet_manager.cc:1398] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:37.771925 10673 ts_tablet_manager.cc:1367] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:58:37.771988 10679 tablet_bootstrap.cc:492] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:37.772542 10673 raft_consensus.cc:348] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.772640 10673 raft_consensus.cc:374] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.772676 10673 raft_consensus.cc:729] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.772778 10679 tablet_bootstrap.cc:654] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:37.772790 10673 consensus_queue.cc:260] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.773087 10673 ts_tablet_manager.cc:1398] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:37.773952 10679 tablet_bootstrap.cc:492] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:37.774032 10679 ts_tablet_manager.cc:1367] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:37.774309 10580 raft_consensus.cc:482] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:37.774382 10580 raft_consensus.cc:504] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.774562 10679 raft_consensus.cc:348] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.774669 10679 raft_consensus.cc:374] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:37.774706 10679 raft_consensus.cc:729] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:37.774787 10580 leader_election.cc:288] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:37.774803 10679 consensus_queue.cc:260] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.775282 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84ae9bb9e84293aa9bfe474279bedc" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:37.775383 9620 raft_consensus.cc:2454] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:37.775636 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84ae9bb9e84293aa9bfe474279bedc" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:37.775797 9356 raft_consensus.cc:2454] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:37.776191 10679 ts_tablet_manager.cc:1398] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:58:37.776223 9159 leader_election.cc:302] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:37.776369 10580 raft_consensus.cc:2781] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:37.776424 10580 raft_consensus.cc:482] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:37.776470 10580 raft_consensus.cc:3037] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.777411 10580 raft_consensus.cc:504] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.777760 10580 leader_election.cc:288] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:37.778017 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84ae9bb9e84293aa9bfe474279bedc" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:37.778048 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84ae9bb9e84293aa9bfe474279bedc" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:37.778098 9356 raft_consensus.cc:3037] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.778143 9620 raft_consensus.cc:3037] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.778805 9356 raft_consensus.cc:2454] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:37.778901 9620 raft_consensus.cc:2454] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:37.779071 9159 leader_election.cc:302] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:37.779222 10580 raft_consensus.cc:2781] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:37.779392 10580 raft_consensus.cc:686] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 LEADER]: Becoming Leader. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Running, Role: LEADER
I20240501 19:58:37.779513 10580 consensus_queue.cc:237] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:37.780763 9077 catalog_manager.cc:5484] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff reported cstate change: term changed from 0 to 1, leader changed from <none> to 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1). New cstate: current_term: 1 leader_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } }
I20240501 19:58:37.873195 10580 raft_consensus.cc:482] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:37.873328 10580 raft_consensus.cc:504] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.873713 10580 leader_election.cc:288] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:37.874080 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d282530a4d442bcbb47f0cbac273f24" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:37.874085 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d282530a4d442bcbb47f0cbac273f24" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:37.874209 9620 raft_consensus.cc:2454] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:37.874224 9356 raft_consensus.cc:2454] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:37.874557 9160 leader_election.cc:302] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:37.874747 10580 raft_consensus.cc:2781] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:37.874794 10580 raft_consensus.cc:482] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:37.874828 10580 raft_consensus.cc:3037] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.875919 10580 raft_consensus.cc:504] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.876411 10580 leader_election.cc:288] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:37.876655 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d282530a4d442bcbb47f0cbac273f24" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:37.876775 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d282530a4d442bcbb47f0cbac273f24" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:37.876812 9620 raft_consensus.cc:3037] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.876873 9356 raft_consensus.cc:3037] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.877813 9620 raft_consensus.cc:2454] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:37.878033 9356 raft_consensus.cc:2454] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:37.878049 9160 leader_election.cc:302] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:37.878188 10580 raft_consensus.cc:2781] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:37.878264 10580 raft_consensus.cc:686] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 LEADER]: Becoming Leader. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Running, Role: LEADER
I20240501 19:58:37.878418 10580 consensus_queue.cc:237] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } }
I20240501 19:58:37.880067 9077 catalog_manager.cc:5484] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff reported cstate change: term changed from 0 to 1, leader changed from <none> to 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1). New cstate: current_term: 1 leader_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } }
I20240501 19:58:37.892530 10576 raft_consensus.cc:482] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:37.892643 10576 raft_consensus.cc:504] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.892966 10576 leader_election.cc:288] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:37.893285 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa70a0afeddd463baef865bf55eadf7f" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:37.893342 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa70a0afeddd463baef865bf55eadf7f" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:37.893415 9620 raft_consensus.cc:2454] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:37.893450 9226 raft_consensus.cc:2454] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:37.893664 9422 leader_election.cc:302] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:37.893808 10576 raft_consensus.cc:2781] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:37.893874 10576 raft_consensus.cc:482] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:37.893915 10576 raft_consensus.cc:3037] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.894613 10576 raft_consensus.cc:504] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.894903 10576 leader_election.cc:288] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:37.895237 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa70a0afeddd463baef865bf55eadf7f" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:37.895236 9620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa70a0afeddd463baef865bf55eadf7f" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:37.895321 9620 raft_consensus.cc:3037] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.895328 9226 raft_consensus.cc:3037] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:37.896119 9226 raft_consensus.cc:2454] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:37.896142 9620 raft_consensus.cc:2454] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:37.896440 9422 leader_election.cc:302] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:37.896672 10576 raft_consensus.cc:2781] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:37.896749 10576 raft_consensus.cc:686] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:37.896867 10576 consensus_queue.cc:237] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:37.898484 9077 catalog_manager.cc:5484] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
W20240501 19:58:38.024104 10689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:38.024310 10689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:38.027447 10689 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 19:58:38.106457 9356 raft_consensus.cc:1261] T 4d282530a4d442bcbb47f0cbac273f24 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer 37fa3f77f5a644d181ccb7ea7ea15fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:38.106550 9620 raft_consensus.cc:1261] T 4d282530a4d442bcbb47f0cbac273f24 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 37fa3f77f5a644d181ccb7ea7ea15fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:38.107316 9225 raft_consensus.cc:1261] T fa70a0afeddd463baef865bf55eadf7f P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:38.107772 10580 consensus_queue.cc:1035] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:38.108069 9356 raft_consensus.cc:1261] T df84ae9bb9e84293aa9bfe474279bedc P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer 37fa3f77f5a644d181ccb7ea7ea15fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:38.108445 9620 raft_consensus.cc:1261] T df84ae9bb9e84293aa9bfe474279bedc P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 37fa3f77f5a644d181ccb7ea7ea15fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:38.108775 10461 consensus_queue.cc:1035] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:38.109612 10685 consensus_queue.cc:1035] T 4d282530a4d442bcbb47f0cbac273f24 P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:38.109894 10685 consensus_queue.cc:1035] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:38.111373 10580 consensus_queue.cc:1035] T df84ae9bb9e84293aa9bfe474279bedc P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:38.114130 9620 raft_consensus.cc:1261] T fa70a0afeddd463baef865bf55eadf7f P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:38.114478 10461 consensus_queue.cc:1035] T fa70a0afeddd463baef865bf55eadf7f P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:38.136281 10649 mvcc.cc:205] Tried to move back new op lower bound from 7022975050165129216 to 7022975048826015744. Current Snapshot: MvccSnapshot[applied={T|T < 7022975050165129216}]
INSERT report
rows total: 2000
time total: 105.604 ms
time per row: 0.0528022 ms
I20240501 19:58:38.173475 9079 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:38.178388 9336 tablet_service.cc:1467] Processing CreateTablet for tablet e47c766f49fd460987950b38ed4896f9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.178439 9204 tablet_service.cc:1467] Processing CreateTablet for tablet e47c766f49fd460987950b38ed4896f9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.178680 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47c766f49fd460987950b38ed4896f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.178685 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47c766f49fd460987950b38ed4896f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.179270 9600 tablet_service.cc:1467] Processing CreateTablet for tablet ce24bd46007b4e8fb15f4537038ced95 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.179471 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce24bd46007b4e8fb15f4537038ced95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.180780 10672 tablet_bootstrap.cc:492] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:38.180938 9205 tablet_service.cc:1467] Processing CreateTablet for tablet ce24bd46007b4e8fb15f4537038ced95 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.181102 10673 tablet_bootstrap.cc:492] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:38.181075 9467 tablet_service.cc:1467] Processing CreateTablet for tablet e47c766f49fd460987950b38ed4896f9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.181155 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce24bd46007b4e8fb15f4537038ced95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.181349 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47c766f49fd460987950b38ed4896f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.181684 10672 tablet_bootstrap.cc:654] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.181937 10673 tablet_bootstrap.cc:654] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.182221 10679 tablet_bootstrap.cc:492] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:38.183070 10672 tablet_bootstrap.cc:492] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:38.183082 10679 tablet_bootstrap.cc:654] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.183151 10672 ts_tablet_manager.cc:1367] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:38.183348 9598 tablet_service.cc:1467] Processing CreateTablet for tablet d96fb2460e6e4dd2a3b8c04067ac4aec (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.183544 9598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96fb2460e6e4dd2a3b8c04067ac4aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.183604 10672 raft_consensus.cc:348] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.183697 10672 raft_consensus.cc:374] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.183728 10672 raft_consensus.cc:729] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.183842 10672 consensus_queue.cc:260] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.184103 10672 ts_tablet_manager.cc:1398] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.184499 10674 tablet_bootstrap.cc:492] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:38.184626 10673 tablet_bootstrap.cc:492] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:38.184693 10673 ts_tablet_manager.cc:1367] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:58:38.185056 10673 raft_consensus.cc:348] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.185145 10673 raft_consensus.cc:374] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.185171 10673 raft_consensus.cc:729] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.185271 10673 consensus_queue.cc:260] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.185350 10674 tablet_bootstrap.cc:654] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.185489 10673 ts_tablet_manager.cc:1398] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:38.186328 10673 tablet_bootstrap.cc:492] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:38.186324 9336 tablet_service.cc:1467] Processing CreateTablet for tablet d96fb2460e6e4dd2a3b8c04067ac4aec (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.186594 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96fb2460e6e4dd2a3b8c04067ac4aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.187316 10673 tablet_bootstrap.cc:654] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.188223 10672 tablet_bootstrap.cc:492] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:38.188328 10673 tablet_bootstrap.cc:492] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:38.188413 10673 ts_tablet_manager.cc:1367] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:38.188843 10673 raft_consensus.cc:348] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.188967 10673 raft_consensus.cc:374] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.189006 10673 raft_consensus.cc:729] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.189118 10673 consensus_queue.cc:260] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.189419 10673 ts_tablet_manager.cc:1398] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.189612 10672 tablet_bootstrap.cc:654] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.190068 9467 tablet_service.cc:1467] Processing CreateTablet for tablet ce24bd46007b4e8fb15f4537038ced95 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.190275 9468 tablet_service.cc:1467] Processing CreateTablet for tablet d96fb2460e6e4dd2a3b8c04067ac4aec (DEFAULT_TABLE table=rebalance_test_table_7 [id=2187665ec7c14ad3a15688aa5520b289]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.190279 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce24bd46007b4e8fb15f4537038ced95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.190770 9468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96fb2460e6e4dd2a3b8c04067ac4aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.193317 10674 tablet_bootstrap.cc:492] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:38.193413 10674 ts_tablet_manager.cc:1367] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:58:38.193660 10672 tablet_bootstrap.cc:492] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:38.193742 10672 ts_tablet_manager.cc:1367] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:58:38.193866 10674 raft_consensus.cc:348] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.193964 10674 raft_consensus.cc:374] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.194003 10674 raft_consensus.cc:729] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.194109 10674 consensus_queue.cc:260] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.194144 10672 raft_consensus.cc:348] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.194232 10672 raft_consensus.cc:374] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.194270 10672 raft_consensus.cc:729] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.194388 10672 consensus_queue.cc:260] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.194671 10672 ts_tablet_manager.cc:1398] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:38.194921 10674 ts_tablet_manager.cc:1398] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.195071 10674 tablet_bootstrap.cc:492] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:38.195765 10674 tablet_bootstrap.cc:654] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.197018 10674 tablet_bootstrap.cc:492] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:38.197103 10674 ts_tablet_manager.cc:1367] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:38.197556 10674 raft_consensus.cc:348] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.197657 10674 raft_consensus.cc:374] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.197696 10674 raft_consensus.cc:729] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.197836 10674 consensus_queue.cc:260] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.198205 10674 ts_tablet_manager.cc:1398] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.198370 10674 tablet_bootstrap.cc:492] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:38.199493 10674 tablet_bootstrap.cc:654] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.200425 10674 tablet_bootstrap.cc:492] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:38.200510 10674 ts_tablet_manager.cc:1367] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:38.200699 10679 tablet_bootstrap.cc:492] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:38.200810 10679 ts_tablet_manager.cc:1367] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.001s
I20240501 19:58:38.200896 10674 raft_consensus.cc:348] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.200990 10674 raft_consensus.cc:374] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.201047 10674 raft_consensus.cc:729] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.201156 10674 consensus_queue.cc:260] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.201277 10679 raft_consensus.cc:348] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.201396 10679 raft_consensus.cc:374] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.201427 10674 ts_tablet_manager.cc:1398] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:38.201439 10679 raft_consensus.cc:729] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.201562 10679 consensus_queue.cc:260] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.201848 10679 ts_tablet_manager.cc:1398] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:58:38.202015 10679 tablet_bootstrap.cc:492] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:38.202850 10679 tablet_bootstrap.cc:654] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.203961 10679 tablet_bootstrap.cc:492] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:38.204035 10679 ts_tablet_manager.cc:1367] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:38.205360 10679 raft_consensus.cc:348] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.205443 10679 raft_consensus.cc:374] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.205466 10679 raft_consensus.cc:729] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.205579 10679 consensus_queue.cc:260] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.205785 10679 ts_tablet_manager.cc:1398] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 19:58:38.211818 10685 raft_consensus.cc:482] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:38.211890 10685 raft_consensus.cc:504] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.212239 10685 leader_election.cc:288] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:38.212519 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47c766f49fd460987950b38ed4896f9" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:38.212625 9488 raft_consensus.cc:2454] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:38.212637 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47c766f49fd460987950b38ed4896f9" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:38.212761 9356 raft_consensus.cc:2454] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 0.
I20240501 19:58:38.212877 9162 leader_election.cc:302] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:38.213021 10685 raft_consensus.cc:2781] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:38.213071 10685 raft_consensus.cc:482] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:38.213097 10685 raft_consensus.cc:3037] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.213770 10685 raft_consensus.cc:504] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.214046 10685 leader_election.cc:288] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Requested vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:38.214287 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47c766f49fd460987950b38ed4896f9" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:38.214362 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47c766f49fd460987950b38ed4896f9" candidate_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:38.214386 9488 raft_consensus.cc:3037] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.214469 9356 raft_consensus.cc:3037] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.215320 9356 raft_consensus.cc:2454] T e47c766f49fd460987950b38ed4896f9 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:38.215325 9488 raft_consensus.cc:2454] T e47c766f49fd460987950b38ed4896f9 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37fa3f77f5a644d181ccb7ea7ea15fff in term 1.
I20240501 19:58:38.215553 9159 leader_election.cc:302] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:38.215709 10685 raft_consensus.cc:2781] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:38.215785 10685 raft_consensus.cc:686] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 LEADER]: Becoming Leader. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Running, Role: LEADER
I20240501 19:58:38.215899 10685 consensus_queue.cc:237] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.217247 9083 catalog_manager.cc:5484] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff reported cstate change: term changed from 0 to 1, leader changed from <none> to 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1). New cstate: current_term: 1 leader_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } }
I20240501 19:58:38.315822 10230 raft_consensus.cc:482] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:38.315938 10230 raft_consensus.cc:504] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.316368 10230 leader_election.cc:288] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:38.316668 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce24bd46007b4e8fb15f4537038ced95" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:38.316735 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce24bd46007b4e8fb15f4537038ced95" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:38.316789 9488 raft_consensus.cc:2454] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 0.
I20240501 19:58:38.316886 9225 raft_consensus.cc:2454] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 0.
I20240501 19:58:38.317062 9556 leader_election.cc:302] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:38.317225 10230 raft_consensus.cc:2781] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:38.317281 10230 raft_consensus.cc:482] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:38.317307 10230 raft_consensus.cc:3037] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.318116 10230 raft_consensus.cc:504] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.318362 10230 leader_election.cc:288] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 election: Requested vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:38.318599 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce24bd46007b4e8fb15f4537038ced95" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:38.318697 9488 raft_consensus.cc:3037] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.318676 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce24bd46007b4e8fb15f4537038ced95" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:38.318786 9225 raft_consensus.cc:3037] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.319478 9488 raft_consensus.cc:2454] T ce24bd46007b4e8fb15f4537038ced95 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 1.
I20240501 19:58:38.319579 9225 raft_consensus.cc:2454] T ce24bd46007b4e8fb15f4537038ced95 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 1.
I20240501 19:58:38.319716 9556 leader_election.cc:302] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:38.319861 10230 raft_consensus.cc:2781] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:38.320031 10230 raft_consensus.cc:686] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 LEADER]: Becoming Leader. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Running, Role: LEADER
I20240501 19:58:38.320165 10230 consensus_queue.cc:237] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.321760 9083 catalog_manager.cc:5484] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 reported cstate change: term changed from 0 to 1, leader changed from <none> to a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "a32eae53a7ae43a9a0295cc52fa16725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } }
I20240501 19:58:38.576253 10576 raft_consensus.cc:482] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:38.576357 10576 raft_consensus.cc:504] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.576716 10576 leader_election.cc:288] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:38.577020 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96fb2460e6e4dd2a3b8c04067ac4aec" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:38.577042 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96fb2460e6e4dd2a3b8c04067ac4aec" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:38.577143 9356 raft_consensus.cc:2454] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:38.577170 9619 raft_consensus.cc:2454] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:38.577416 9421 leader_election.cc:302] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:38.577562 10576 raft_consensus.cc:2781] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:38.577610 10576 raft_consensus.cc:482] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:38.577656 10576 raft_consensus.cc:3037] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.578374 10576 raft_consensus.cc:504] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.578655 10576 leader_election.cc:288] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:38.578917 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96fb2460e6e4dd2a3b8c04067ac4aec" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:38.578953 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96fb2460e6e4dd2a3b8c04067ac4aec" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:38.579006 9356 raft_consensus.cc:3037] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.579097 9619 raft_consensus.cc:3037] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.579777 9356 raft_consensus.cc:2454] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:38.579905 9619 raft_consensus.cc:2454] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:38.580021 9421 leader_election.cc:302] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:38.580188 10576 raft_consensus.cc:2781] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:38.580281 10576 raft_consensus.cc:686] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:38.580415 10576 consensus_queue.cc:237] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.582012 9083 catalog_manager.cc:5484] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } }
I20240501 19:58:38.660959 10685 consensus_queue.cc:1035] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:38.665918 10685 consensus_queue.cc:1035] T e47c766f49fd460987950b38ed4896f9 P 37fa3f77f5a644d181ccb7ea7ea15fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:58:38.747648 10740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:38.747855 10740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 19:58:38.750300 10230 consensus_queue.cc:1035] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:58:38.750774 10740 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:58:38.753166 10230 consensus_queue.cc:1035] T ce24bd46007b4e8fb15f4537038ced95 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240501 19:58:38.850101 9619 raft_consensus.cc:1261] T d96fb2460e6e4dd2a3b8c04067ac4aec P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:38.852239 10461 consensus_queue.cc:1035] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:38.852870 9357 raft_consensus.cc:1261] T d96fb2460e6e4dd2a3b8c04067ac4aec P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:38.853225 10576 consensus_queue.cc:1035] T d96fb2460e6e4dd2a3b8c04067ac4aec P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.651 ms
time per row: 0.0578256 ms
I20240501 19:58:38.910493 9075 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:38.915228 9205 tablet_service.cc:1467] Processing CreateTablet for tablet 7c85520939a14f0bae6e6419e29ecce5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.915369 9336 tablet_service.cc:1467] Processing CreateTablet for tablet 7c85520939a14f0bae6e6419e29ecce5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.915505 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c85520939a14f0bae6e6419e29ecce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.915578 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c85520939a14f0bae6e6419e29ecce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.916196 9206 tablet_service.cc:1467] Processing CreateTablet for tablet fb15b158e91d42f5a993a53bffed32b6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.916285 9335 tablet_service.cc:1467] Processing CreateTablet for tablet c28e44917cf44eab9118dda1f82991d0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.916410 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb15b158e91d42f5a993a53bffed32b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.916473 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28e44917cf44eab9118dda1f82991d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.917649 9204 tablet_service.cc:1467] Processing CreateTablet for tablet c28e44917cf44eab9118dda1f82991d0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.917809 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28e44917cf44eab9118dda1f82991d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.918658 9598 tablet_service.cc:1467] Processing CreateTablet for tablet c28e44917cf44eab9118dda1f82991d0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.918908 9598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28e44917cf44eab9118dda1f82991d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.919108 9467 tablet_service.cc:1467] Processing CreateTablet for tablet fb15b158e91d42f5a993a53bffed32b6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.919337 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb15b158e91d42f5a993a53bffed32b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.919495 10801 tablet_bootstrap.cc:492] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:38.919106 9468 tablet_service.cc:1467] Processing CreateTablet for tablet 7c85520939a14f0bae6e6419e29ecce5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.919750 9468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c85520939a14f0bae6e6419e29ecce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.920531 10801 tablet_bootstrap.cc:654] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.921265 10802 tablet_bootstrap.cc:492] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:38.918658 9600 tablet_service.cc:1467] Processing CreateTablet for tablet fb15b158e91d42f5a993a53bffed32b6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=10d0ea627f4740b3886dfe626a36ca3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:38.921435 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb15b158e91d42f5a993a53bffed32b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:38.921792 10803 tablet_bootstrap.cc:492] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:38.922220 10802 tablet_bootstrap.cc:654] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.922688 10803 tablet_bootstrap.cc:654] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.923056 10805 tablet_bootstrap.cc:492] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:38.923499 10802 tablet_bootstrap.cc:492] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:38.923615 10802 ts_tablet_manager.cc:1367] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:38.923727 10803 tablet_bootstrap.cc:492] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:38.923807 10803 ts_tablet_manager.cc:1367] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:38.923990 10805 tablet_bootstrap.cc:654] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.924065 10802 raft_consensus.cc:348] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:38.924173 10802 raft_consensus.cc:374] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.924186 10803 raft_consensus.cc:348] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.924283 10803 raft_consensus.cc:374] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.924320 10803 raft_consensus.cc:729] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.924445 10803 consensus_queue.cc:260] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.924752 10803 ts_tablet_manager.cc:1398] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.924872 10802 raft_consensus.cc:729] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.924970 10803 tablet_bootstrap.cc:492] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:38.925006 10802 consensus_queue.cc:260] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:38.925105 10801 tablet_bootstrap.cc:492] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:38.925184 10801 ts_tablet_manager.cc:1367] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:58:38.925256 10802 ts_tablet_manager.cc:1398] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:38.925410 10802 tablet_bootstrap.cc:492] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:38.925616 10801 raft_consensus.cc:348] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:38.925704 10803 tablet_bootstrap.cc:654] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.925714 10801 raft_consensus.cc:374] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.925747 10801 raft_consensus.cc:729] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.925851 10801 consensus_queue.cc:260] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:38.926141 10801 ts_tablet_manager.cc:1398] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:38.926285 10801 tablet_bootstrap.cc:492] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:38.926344 10802 tablet_bootstrap.cc:654] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.927130 10801 tablet_bootstrap.cc:654] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.928148 10805 tablet_bootstrap.cc:492] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:38.928225 10805 ts_tablet_manager.cc:1367] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:58:38.928696 10805 raft_consensus.cc:348] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.928797 10805 raft_consensus.cc:374] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.928829 10805 raft_consensus.cc:729] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.928948 10805 consensus_queue.cc:260] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.929096 10803 tablet_bootstrap.cc:492] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:38.929183 10803 ts_tablet_manager.cc:1367] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:58:38.929489 10803 raft_consensus.cc:348] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.929565 10805 ts_tablet_manager.cc:1398] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.929585 10803 raft_consensus.cc:374] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.929622 10803 raft_consensus.cc:729] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.929740 10805 tablet_bootstrap.cc:492] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:38.929751 10803 consensus_queue.cc:260] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.930009 10803 ts_tablet_manager.cc:1398] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:38.930667 10801 tablet_bootstrap.cc:492] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:38.930743 10801 ts_tablet_manager.cc:1367] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:58:38.931121 10805 tablet_bootstrap.cc:654] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.931185 10801 raft_consensus.cc:348] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.931293 10801 raft_consensus.cc:374] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.931335 10801 raft_consensus.cc:729] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.931454 10801 consensus_queue.cc:260] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.931710 10801 ts_tablet_manager.cc:1398] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:58:38.931936 10801 tablet_bootstrap.cc:492] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:38.932687 10802 tablet_bootstrap.cc:492] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:38.932761 10802 ts_tablet_manager.cc:1367] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:58:38.933156 10801 tablet_bootstrap.cc:654] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:38.933169 10802 raft_consensus.cc:348] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.933271 10802 raft_consensus.cc:374] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.933313 10802 raft_consensus.cc:729] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.933452 10802 consensus_queue.cc:260] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.933750 10802 ts_tablet_manager.cc:1398] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.934192 10805 tablet_bootstrap.cc:492] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:38.934269 10805 ts_tablet_manager.cc:1367] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:58:38.934368 10801 tablet_bootstrap.cc:492] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:38.934430 10801 ts_tablet_manager.cc:1367] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:58:38.934654 10805 raft_consensus.cc:348] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:38.934743 10805 raft_consensus.cc:374] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.934782 10805 raft_consensus.cc:729] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.934787 10801 raft_consensus.cc:348] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.934899 10801 raft_consensus.cc:374] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:38.934935 10801 raft_consensus.cc:729] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:38.934913 10805 consensus_queue.cc:260] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:38.935062 10801 consensus_queue.cc:260] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:38.935178 10805 ts_tablet_manager.cc:1398] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.935304 10801 ts_tablet_manager.cc:1398] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:38.945060 10576 raft_consensus.cc:482] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:38.945130 10576 raft_consensus.cc:504] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.945451 10576 leader_election.cc:288] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:38.945699 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15b158e91d42f5a993a53bffed32b6" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:38.945756 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15b158e91d42f5a993a53bffed32b6" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:38.945788 9619 raft_consensus.cc:2454] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:38.945865 9225 raft_consensus.cc:2454] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 0.
I20240501 19:58:38.946004 9422 leader_election.cc:302] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:38.946154 10576 raft_consensus.cc:2781] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:38.946205 10576 raft_consensus.cc:482] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:38.946231 10576 raft_consensus.cc:3037] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.946909 10576 raft_consensus.cc:504] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.947221 10576 leader_election.cc:288] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:38.947499 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15b158e91d42f5a993a53bffed32b6" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:38.947530 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15b158e91d42f5a993a53bffed32b6" candidate_uuid: "717a0743a9b741f7909ac2557611bb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:38.947609 9225 raft_consensus.cc:3037] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.947598 9619 raft_consensus.cc:3037] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.948418 9619 raft_consensus.cc:2454] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:38.948413 9225 raft_consensus.cc:2454] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717a0743a9b741f7909ac2557611bb26 in term 1.
I20240501 19:58:38.948685 9422 leader_election.cc:302] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 717a0743a9b741f7909ac2557611bb26; no voters:
I20240501 19:58:38.948828 10576 raft_consensus.cc:2781] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:38.948910 10576 raft_consensus.cc:686] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Leader. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:38.949026 10576 consensus_queue.cc:237] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:38.950443 9075 catalog_manager.cc:5484] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717a0743a9b741f7909ac2557611bb26 (127.17.65.3). New cstate: current_term: 1 leader_uuid: "717a0743a9b741f7909ac2557611bb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } }
I20240501 19:58:38.996639 10586 raft_consensus.cc:482] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:38.996728 10586 raft_consensus.cc:504] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:38.997036 10586 leader_election.cc:288] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:38.997274 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c85520939a14f0bae6e6419e29ecce5" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:38.997319 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c85520939a14f0bae6e6419e29ecce5" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:38.997406 9225 raft_consensus.cc:2454] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:38.997419 9488 raft_consensus.cc:2454] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:38.997648 9291 leader_election.cc:302] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:38.997793 10586 raft_consensus.cc:2781] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:38.997855 10586 raft_consensus.cc:482] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:38.997882 10586 raft_consensus.cc:3037] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.998677 10586 raft_consensus.cc:504] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:38.998943 10586 leader_election.cc:288] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:38.999250 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c85520939a14f0bae6e6419e29ecce5" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:38.999315 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c85520939a14f0bae6e6419e29ecce5" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:38.999368 9488 raft_consensus.cc:3037] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:38.999409 9225 raft_consensus.cc:3037] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.000281 9488 raft_consensus.cc:2454] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:39.000281 9225 raft_consensus.cc:2454] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:39.000566 9293 leader_election.cc:302] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:39.000762 10586 raft_consensus.cc:2781] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:39.000839 10586 raft_consensus.cc:686] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:39.000967 10586 consensus_queue.cc:237] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.002394 9075 catalog_manager.cc:5484] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
I20240501 19:58:39.128753 10230 raft_consensus.cc:482] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:39.128852 10230 raft_consensus.cc:504] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.129210 10230 leader_election.cc:288] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:39.129508 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e44917cf44eab9118dda1f82991d0" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:39.129621 9225 raft_consensus.cc:2454] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 0.
I20240501 19:58:39.129873 9554 leader_election.cc:302] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:39.130046 10230 raft_consensus.cc:2781] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:39.130112 10230 raft_consensus.cc:482] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:39.130151 10230 raft_consensus.cc:3037] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.131062 10230 raft_consensus.cc:504] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.131382 10230 leader_election.cc:288] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481)
I20240501 19:58:39.131649 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e44917cf44eab9118dda1f82991d0" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:39.131758 9225 raft_consensus.cc:3037] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.132637 9225 raft_consensus.cc:2454] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 1.
I20240501 19:58:39.132889 9554 leader_election.cc:302] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:39.132936 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e44917cf44eab9118dda1f82991d0" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15" is_pre_election: true
I20240501 19:58:39.132977 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e44917cf44eab9118dda1f82991d0" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:39.133055 10230 raft_consensus.cc:2781] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:39.133077 9357 raft_consensus.cc:2454] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 0.
I20240501 19:58:39.133138 10230 raft_consensus.cc:686] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 LEADER]: Becoming Leader. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Running, Role: LEADER
I20240501 19:58:39.133273 10230 consensus_queue.cc:237] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.134853 9075 catalog_manager.cc:5484] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 reported cstate change: term changed from 0 to 1, leader changed from <none> to a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4). New cstate: current_term: 1 leader_uuid: "a32eae53a7ae43a9a0295cc52fa16725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } }
W20240501 19:58:39.254891 10820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:39.255107 10820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:39.257743 10820 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 19:58:39.327471 9488 raft_consensus.cc:1261] T 7c85520939a14f0bae6e6419e29ecce5 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.329084 9225 raft_consensus.cc:1261] T 7c85520939a14f0bae6e6419e29ecce5 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.329497 10815 consensus_queue.cc:1035] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.331393 9225 raft_consensus.cc:1261] T c28e44917cf44eab9118dda1f82991d0 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer a32eae53a7ae43a9a0295cc52fa16725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.331883 10230 consensus_queue.cc:1035] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.334621 9357 raft_consensus.cc:3037] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.335695 9357 raft_consensus.cc:1261] T c28e44917cf44eab9118dda1f82991d0 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Refusing update from remote peer a32eae53a7ae43a9a0295cc52fa16725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.336679 10737 consensus_queue.cc:1035] T c28e44917cf44eab9118dda1f82991d0 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.338955 10815 consensus_queue.cc:1035] T 7c85520939a14f0bae6e6419e29ecce5 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.344230 9226 raft_consensus.cc:1261] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.344349 9619 raft_consensus.cc:1261] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 717a0743a9b741f7909ac2557611bb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.344758 10461 consensus_queue.cc:1035] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.344991 10461 consensus_queue.cc:1035] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.065 ms
time per row: 0.0525325 ms
I20240501 19:58:39.401587 9074 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:58:39.406967 9335 tablet_service.cc:1467] Processing CreateTablet for tablet d04c802891964eaf8a73db2e807a7bf2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.407335 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04c802891964eaf8a73db2e807a7bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.407559 9600 tablet_service.cc:1467] Processing CreateTablet for tablet d04c802891964eaf8a73db2e807a7bf2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.407814 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04c802891964eaf8a73db2e807a7bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.409008 10802 tablet_bootstrap.cc:492] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:39.409238 9204 tablet_service.cc:1467] Processing CreateTablet for tablet 831f4c1c3d8e4653a20db124d656a572 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.409473 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831f4c1c3d8e4653a20db124d656a572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.409809 10802 tablet_bootstrap.cc:654] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.409770 9598 tablet_service.cc:1467] Processing CreateTablet for tablet b16a1ce3e3c14323b261d9cbd4016a30 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.409956 9598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16a1ce3e3c14323b261d9cbd4016a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.410745 10802 tablet_bootstrap.cc:492] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:39.410816 10802 ts_tablet_manager.cc:1367] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:39.411234 10802 raft_consensus.cc:348] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.411337 10802 raft_consensus.cc:374] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.411367 10802 raft_consensus.cc:729] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.411489 10802 consensus_queue.cc:260] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.411620 10803 tablet_bootstrap.cc:492] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:39.411758 10802 ts_tablet_manager.cc:1398] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:39.411895 9467 tablet_service.cc:1467] Processing CreateTablet for tablet b16a1ce3e3c14323b261d9cbd4016a30 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.412123 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16a1ce3e3c14323b261d9cbd4016a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.412307 10803 tablet_bootstrap.cc:654] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.412820 10801 tablet_bootstrap.cc:492] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff: Bootstrap starting.
I20240501 19:58:39.413422 10801 tablet_bootstrap.cc:654] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.413954 10805 tablet_bootstrap.cc:492] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:39.414110 9468 tablet_service.cc:1467] Processing CreateTablet for tablet d04c802891964eaf8a73db2e807a7bf2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.414147 9466 tablet_service.cc:1467] Processing CreateTablet for tablet 831f4c1c3d8e4653a20db124d656a572 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.414271 9468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04c802891964eaf8a73db2e807a7bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.414458 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831f4c1c3d8e4653a20db124d656a572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.414721 10805 tablet_bootstrap.cc:654] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.417862 9336 tablet_service.cc:1467] Processing CreateTablet for tablet 831f4c1c3d8e4653a20db124d656a572 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.418102 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831f4c1c3d8e4653a20db124d656a572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.418314 10801 tablet_bootstrap.cc:492] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff: No bootstrap required, opened a new log
I20240501 19:58:39.418406 10801 ts_tablet_manager.cc:1367] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:58:39.418782 10801 raft_consensus.cc:348] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.418895 10801 raft_consensus.cc:374] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.418934 10801 raft_consensus.cc:729] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fa3f77f5a644d181ccb7ea7ea15fff, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.419078 10801 consensus_queue.cc:260] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.419358 10801 ts_tablet_manager.cc:1398] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:39.420069 10802 tablet_bootstrap.cc:492] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:39.420830 10802 tablet_bootstrap.cc:654] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.421401 10803 tablet_bootstrap.cc:492] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:39.421478 10803 ts_tablet_manager.cc:1367] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:58:39.422014 10802 tablet_bootstrap.cc:492] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:39.422012 10803 raft_consensus.cc:348] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.422092 10802 ts_tablet_manager.cc:1367] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:39.422114 10803 raft_consensus.cc:374] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.422151 10803 raft_consensus.cc:729] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.422302 10803 consensus_queue.cc:260] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.422561 10803 ts_tablet_manager.cc:1398] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:39.422554 10802 raft_consensus.cc:348] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.422677 10802 raft_consensus.cc:374] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.422717 10802 raft_consensus.cc:729] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.422719 10803 tablet_bootstrap.cc:492] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725: Bootstrap starting.
I20240501 19:58:39.422822 10802 consensus_queue.cc:260] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.423131 10802 ts_tablet_manager.cc:1398] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:39.423576 10803 tablet_bootstrap.cc:654] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.424767 9335 tablet_service.cc:1467] Processing CreateTablet for tablet b16a1ce3e3c14323b261d9cbd4016a30 (DEFAULT_TABLE table=rebalance_test_table_9 [id=8c9fabb173214be9bfea459a57340402]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:58:39.424966 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16a1ce3e3c14323b261d9cbd4016a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:58:39.426800 10802 tablet_bootstrap.cc:492] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15: Bootstrap starting.
I20240501 19:58:39.427290 10803 tablet_bootstrap.cc:492] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725: No bootstrap required, opened a new log
I20240501 19:58:39.427378 10803 ts_tablet_manager.cc:1367] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:58:39.427593 10802 tablet_bootstrap.cc:654] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.427829 10803 raft_consensus.cc:348] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.427913 10803 raft_consensus.cc:374] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.427947 10803 raft_consensus.cc:729] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.428056 10803 consensus_queue.cc:260] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.428285 10803 ts_tablet_manager.cc:1398] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:39.428310 10805 tablet_bootstrap.cc:492] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:39.428395 10805 ts_tablet_manager.cc:1367] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 19:58:39.428819 10805 raft_consensus.cc:348] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.428917 10805 raft_consensus.cc:374] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.428941 10805 raft_consensus.cc:729] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.429040 10805 consensus_queue.cc:260] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.429276 10805 ts_tablet_manager.cc:1398] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:39.429502 10805 tablet_bootstrap.cc:492] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:39.430150 10802 tablet_bootstrap.cc:492] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15: No bootstrap required, opened a new log
I20240501 19:58:39.430222 10802 ts_tablet_manager.cc:1367] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:58:39.430394 10805 tablet_bootstrap.cc:654] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.430812 10802 raft_consensus.cc:348] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.430900 10802 raft_consensus.cc:374] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.430928 10802 raft_consensus.cc:729] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.431049 10802 consensus_queue.cc:260] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.431329 10802 ts_tablet_manager.cc:1398] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:39.431571 10805 tablet_bootstrap.cc:492] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:39.431645 10805 ts_tablet_manager.cc:1367] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:39.431999 10805 raft_consensus.cc:348] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.432104 10805 raft_consensus.cc:374] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.432133 10805 raft_consensus.cc:729] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.432296 10805 consensus_queue.cc:260] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.432610 10805 ts_tablet_manager.cc:1398] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:58:39.433282 10805 tablet_bootstrap.cc:492] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26: Bootstrap starting.
I20240501 19:58:39.434085 10805 tablet_bootstrap.cc:654] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26: Neither blocks nor log segments found. Creating new log.
I20240501 19:58:39.435180 10805 tablet_bootstrap.cc:492] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26: No bootstrap required, opened a new log
I20240501 19:58:39.435261 10805 ts_tablet_manager.cc:1367] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:39.435757 10805 raft_consensus.cc:348] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.435861 10805 raft_consensus.cc:374] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:58:39.435891 10805 raft_consensus.cc:729] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Initialized, Role: FOLLOWER
I20240501 19:58:39.436013 10805 consensus_queue.cc:260] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.436271 10805 ts_tablet_manager.cc:1398] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:58:39.468797 10815 raft_consensus.cc:482] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:39.468899 10815 raft_consensus.cc:504] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.469228 10815 leader_election.cc:288] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:39.469528 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a1ce3e3c14323b261d9cbd4016a30" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:39.469528 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a1ce3e3c14323b261d9cbd4016a30" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:39.469653 9619 raft_consensus.cc:2454] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:39.469663 9488 raft_consensus.cc:2454] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:39.469918 9291 leader_election.cc:302] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:39.470059 10815 raft_consensus.cc:2781] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:39.470113 10815 raft_consensus.cc:482] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:39.470136 10815 raft_consensus.cc:3037] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.470897 10815 raft_consensus.cc:504] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.471181 10815 leader_election.cc:288] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:39.471233 10586 raft_consensus.cc:482] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:39.471290 10586 raft_consensus.cc:504] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.471489 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a1ce3e3c14323b261d9cbd4016a30" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:39.471509 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a1ce3e3c14323b261d9cbd4016a30" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:39.471586 10586 leader_election.cc:288] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:39.471596 9488 raft_consensus.cc:3037] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.471596 9619 raft_consensus.cc:3037] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.471917 9489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831f4c1c3d8e4653a20db124d656a572" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:39.472000 9489 raft_consensus.cc:2454] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:39.472204 9293 leader_election.cc:302] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:39.472345 10586 raft_consensus.cc:2781] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:39.472388 10586 raft_consensus.cc:482] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:39.472419 10586 raft_consensus.cc:3037] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.472482 9488 raft_consensus.cc:2454] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:39.472482 9619 raft_consensus.cc:2454] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:39.472723 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831f4c1c3d8e4653a20db124d656a572" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" is_pre_election: true
I20240501 19:58:39.472777 9291 leader_election.cc:302] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:39.472862 9225 raft_consensus.cc:2454] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:39.472918 10815 raft_consensus.cc:2781] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:39.473057 10815 raft_consensus.cc:686] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:39.473253 10586 raft_consensus.cc:504] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.473229 10815 consensus_queue.cc:237] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.473554 10586 leader_election.cc:288] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:39.473990 9225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831f4c1c3d8e4653a20db124d656a572" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:39.474083 9225 raft_consensus.cc:3037] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.474819 9225 raft_consensus.cc:2454] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:39.474890 9074 catalog_manager.cc:5484] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
I20240501 19:58:39.474993 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831f4c1c3d8e4653a20db124d656a572" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:39.475106 9488 raft_consensus.cc:3037] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.475116 9291 leader_election.cc:302] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:39.475262 10815 raft_consensus.cc:2781] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:39.475337 10815 raft_consensus.cc:686] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:39.475463 10815 consensus_queue.cc:237] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:39.475879 9488 raft_consensus.cc:2454] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:39.478302 9074 catalog_manager.cc:5484] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
I20240501 19:58:39.538609 10866 raft_consensus.cc:482] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:39.538730 10866 raft_consensus.cc:504] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.539099 10866 leader_election.cc:288] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:39.539350 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c802891964eaf8a73db2e807a7bf2" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725" is_pre_election: true
I20240501 19:58:39.539348 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c802891964eaf8a73db2e807a7bf2" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26" is_pre_election: true
I20240501 19:58:39.539463 9488 raft_consensus.cc:2454] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:39.539472 9619 raft_consensus.cc:2454] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 0.
I20240501 19:58:39.539695 9291 leader_election.cc:302] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:39.539870 10866 raft_consensus.cc:2781] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:58:39.539924 10866 raft_consensus.cc:482] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:58:39.539963 10866 raft_consensus.cc:3037] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.540755 10866 raft_consensus.cc:504] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.541030 10866 leader_election.cc:288] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Requested vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4:45367)
I20240501 19:58:39.541280 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c802891964eaf8a73db2e807a7bf2" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:39.541332 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c802891964eaf8a73db2e807a7bf2" candidate_uuid: "7b8ab321c9ce4518a42b35047ae72c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
I20240501 19:58:39.541388 9488 raft_consensus.cc:3037] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.541425 9619 raft_consensus.cc:3037] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:58:39.542064 9488 raft_consensus.cc:2454] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:39.542310 9619 raft_consensus.cc:2454] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b8ab321c9ce4518a42b35047ae72c15 in term 1.
I20240501 19:58:39.542402 9293 leader_election.cc:302] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717a0743a9b741f7909ac2557611bb26, 7b8ab321c9ce4518a42b35047ae72c15; no voters:
I20240501 19:58:39.542572 10866 raft_consensus.cc:2781] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:58:39.542652 10866 raft_consensus.cc:686] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:39.542950 10866 consensus_queue.cc:237] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:39.544575 9074 catalog_manager.cc:5484] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2). New cstate: current_term: 1 leader_uuid: "7b8ab321c9ce4518a42b35047ae72c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } }
W20240501 19:58:39.667416 10870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:39.667613 10870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:39.670351 10870 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 19:58:39.756726 9488 raft_consensus.cc:1261] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.757148 9619 raft_consensus.cc:1261] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.757141 10866 consensus_queue.cc:1035] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.759613 10815 consensus_queue.cc:1035] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.761185 9226 raft_consensus.cc:1261] T 831f4c1c3d8e4653a20db124d656a572 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.761708 9620 raft_consensus.cc:1261] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.762140 10815 consensus_queue.cc:1035] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.762972 9487 raft_consensus.cc:1261] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.763379 10586 consensus_queue.cc:1035] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.763691 10586 consensus_queue.cc:1035] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:58:39.764377 9489 raft_consensus.cc:1261] T 831f4c1c3d8e4653a20db124d656a572 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Refusing update from remote peer 7b8ab321c9ce4518a42b35047ae72c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:58:39.767186 10815 consensus_queue.cc:1035] T 831f4c1c3d8e4653a20db124d656a572 P 7b8ab321c9ce4518a42b35047ae72c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.682 ms
time per row: 0.055841 ms
I20240501 19:58:39.820263 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.5:36591
--local_ip_for_outbound_sockets=127.17.65.5
--tserver_master_addrs=127.17.65.62:45861
--webserver_port=39509
--webserver_interface=127.17.65.5
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:58:39.931823 10904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:39.931999 10904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:39.932054 10904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:39.935102 10904 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:39.935156 10904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:39.935240 10904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.5
I20240501 19:58:39.938591 10904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.5:36591
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.65.5
--webserver_port=39509
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.10904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.5
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:39.939391 10904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:39.940429 10904 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:39.948835 10912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:39.949353 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:39.949980 10909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:39.951081 10904 server_base.cc:1034] running on GCE node
I20240501 19:58:39.951411 10904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:39.951937 10904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:39.953100 10904 hybrid_clock.cc:648] HybridClock initialized: now 1714593519953093 us; error 26 us; skew 500 ppm
I20240501 19:58:39.961061 10904 webserver.cc:460] Webserver started at http://127.17.65.5:39509/ using document root <none> and password file <none>
I20240501 19:58:39.961551 10904 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:39.961630 10904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:39.964841 10904 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 19:58:39.966496 10918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:39.967461 10904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 19:58:39.967559 10904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8ef448fac22b4340ae5d5155c54d6e9f"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:39.967921 10904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:39.978778 10904 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:39.979252 10904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:39.979398 10904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:39.979813 10904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:39.980572 10904 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:39.980614 10904 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:39.980652 10904 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:39.980681 10904 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:39.988896 10904 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.5:36591
I20240501 19:58:39.988919 11031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.5:36591 every 8 connection(s)
I20240501 19:58:39.989748 10904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:58:39.995234 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 10904
I20240501 19:58:39.995728 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.6:38277
--local_ip_for_outbound_sockets=127.17.65.6
--tserver_master_addrs=127.17.65.62:45861
--webserver_port=45757
--webserver_interface=127.17.65.6
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:39.997781 11032 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:39.997956 11032 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:39.998307 11032 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:39.999208 9076 ts_manager.cc:194] Re-registered known tserver with Master: 8ef448fac22b4340ae5d5155c54d6e9f (127.17.65.5:36591)
I20240501 19:58:39.999758 9076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.5:32967
W20240501 19:58:40.097061 11035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:40.097249 11035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:40.097295 11035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:40.100338 11035 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:40.100392 11035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:40.100451 11035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.6
I20240501 19:58:40.103986 11035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.6:38277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.65.6
--webserver_port=45757
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.11035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.6
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:40.104701 11035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:40.105664 11035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:40.113787 11043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:40.114490 11040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:40.115226 11041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:40.115958 11035 server_base.cc:1034] running on GCE node
I20240501 19:58:40.116284 11035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:40.116744 11035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:40.117882 11035 hybrid_clock.cc:648] HybridClock initialized: now 1714593520117870 us; error 25 us; skew 500 ppm
I20240501 19:58:40.119583 11035 webserver.cc:460] Webserver started at http://127.17.65.6:45757/ using document root <none> and password file <none>
I20240501 19:58:40.120069 11035 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:40.120142 11035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:40.123423 11035 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:58:40.125174 11049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.126058 11035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:58:40.126154 11035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "929f8cb9b40f480bbaff000ef35968d8"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:40.126540 11035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:40.139950 11035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:40.140419 11035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:40.140554 11035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:40.140990 11035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:40.141748 11035 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:40.141793 11035 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.141844 11035 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:40.141870 11035 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.150632 11035 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.6:38277
I20240501 19:58:40.150665 11162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.6:38277 every 8 connection(s)
I20240501 19:58:40.151567 11035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:58:40.159164 11163 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:40.159366 11163 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:40.159755 11163 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:40.160601 9076 ts_manager.cc:194] Re-registered known tserver with Master: 929f8cb9b40f480bbaff000ef35968d8 (127.17.65.6:38277)
I20240501 19:58:40.161090 9076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.6:41469
I20240501 19:58:40.161104 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 11035
I20240501 19:58:40.161505 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.7:39703
--local_ip_for_outbound_sockets=127.17.65.7
--tserver_master_addrs=127.17.65.62:45861
--webserver_port=40049
--webserver_interface=127.17.65.7
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:58:40.264432 11166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:40.264618 11166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:40.264673 11166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:40.267446 11166 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:40.267498 11166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:40.267583 11166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.7
I20240501 19:58:40.270968 11166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.7:39703
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.65.7
--webserver_port=40049
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.11166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.7
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:40.271742 11166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:40.272815 11166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:40.280766 11171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:40.281440 11172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:40.282029 11174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:40.283087 11166 server_base.cc:1034] running on GCE node
I20240501 19:58:40.283442 11166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:40.283926 11166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:40.285049 11166 hybrid_clock.cc:648] HybridClock initialized: now 1714593520285045 us; error 32 us; skew 500 ppm
I20240501 19:58:40.286675 11166 webserver.cc:460] Webserver started at http://127.17.65.7:40049/ using document root <none> and password file <none>
I20240501 19:58:40.287197 11166 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:40.287289 11166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:40.290376 11166 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 19:58:40.292018 11180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.292919 11166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:40.293011 11166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fc80d0dfc6bd41ecb402d108c9a393ca"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:40.293390 11166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:40.302446 11166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:40.302899 11166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:40.303052 11166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:40.303479 11166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:40.304195 11166 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:40.304236 11166 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.304282 11166 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:40.304303 11166 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.312137 11166 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.7:39703
I20240501 19:58:40.312158 11293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.7:39703 every 8 connection(s)
I20240501 19:58:40.313205 11166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:58:40.316736 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 11166
I20240501 19:58:40.317152 17668 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3cEqqr/build/debug/bin/kudu
/tmp/dist-test-task3cEqqr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.65.8:37673
--local_ip_for_outbound_sockets=127.17.65.8
--tserver_master_addrs=127.17.65.62:45861
--webserver_port=45667
--webserver_interface=127.17.65.8
--builtin_ntp_servers=127.17.65.20:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:58:40.321717 11294 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:40.321897 11294 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:40.322294 11294 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:40.323213 9076 ts_manager.cc:194] Re-registered known tserver with Master: fc80d0dfc6bd41ecb402d108c9a393ca (127.17.65.7:39703)
I20240501 19:58:40.323726 9076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.7:41815
W20240501 19:58:40.421859 11297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:58:40.422047 11297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:58:40.422109 11297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:58:40.425097 11297 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:58:40.425164 11297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:58:40.425244 11297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.65.8
I20240501 19:58:40.428614 11297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.65.20:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.65.8:37673
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.17.65.8
--webserver_port=45667
--tserver_master_addrs=127.17.65.62:45861
--never_fsync=true
--heap_profile_path=/tmp/kudu.11297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.65.8
--log_dir=/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:58:40.429380 11297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:58:40.430369 11297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:58:40.438750 11306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:40.439409 11303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:58:40.440399 11304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:58:40.441223 11297 server_base.cc:1034] running on GCE node
I20240501 19:58:40.441532 11297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:58:40.441982 11297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:58:40.443087 11297 hybrid_clock.cc:648] HybridClock initialized: now 1714593520443078 us; error 21 us; skew 500 ppm
I20240501 19:58:40.444744 11297 webserver.cc:460] Webserver started at http://127.17.65.8:45667/ using document root <none> and password file <none>
I20240501 19:58:40.445217 11297 fs_manager.cc:359] Metadata directory not provided
I20240501 19:58:40.445289 11297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:58:40.448328 11297 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:58:40.449995 11312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.450819 11297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:58:40.450912 11297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d12a830c42564783a28de6d174f48721"
format_stamp: "Formatted at 2024-05-01 19:58:30 on dist-test-slave-703j"
I20240501 19:58:40.451280 11297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:58:40.460657 11297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:58:40.461073 11297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:58:40.461202 11297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:58:40.461620 11297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:58:40.462338 11297 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:58:40.462412 11297 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.462463 11297 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:58:40.462492 11297 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:58:40.470427 11297 rpc_server.cc:307] RPC server started. Bound to: 127.17.65.8:37673
I20240501 19:58:40.470610 11425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.65.8:37673 every 8 connection(s)
I20240501 19:58:40.471341 11297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3cEqqr/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714593353055615-17668-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:58:40.472564 17668 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3cEqqr/build/debug/bin/kudu as pid 11297
I20240501 19:58:40.472810 17668 test_util.cc:268] Using random seed: -373763351
I20240501 19:58:40.481523 11426 heartbeater.cc:347] Connected to a master server at 127.17.65.62:45861
I20240501 19:58:40.481896 11426 heartbeater.cc:464] Registering TS with master...
I20240501 19:58:40.482352 11426 heartbeater.cc:511] Master 127.17.65.62:45861 requested a full tablet report, sending...
I20240501 19:58:40.483213 9074 ts_manager.cc:194] Re-registered known tserver with Master: d12a830c42564783a28de6d174f48721 (127.17.65.8:37673)
I20240501 19:58:40.483878 9074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.65.8:58519
I20240501 19:58:40.484997 17668 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 19:58:40.485193 17668 test_util.cc:268] Using random seed: -373750969
I20240501 19:58:40.495911 17668 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 19:58:40.496029 17668 test_util.cc:268] Using random seed: -373740132
I20240501 19:58:40.505270 17668 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 19:58:40.505409 17668 test_util.cc:268] Using random seed: -373730752
I20240501 19:58:40.511952 9620 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b16a1ce3e3c14323b261d9cbd4016a30"
dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
from {username='slave'} at 127.0.0.1:43300
I20240501 19:58:40.512115 9620 raft_consensus.cc:482] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:58:40.512180 9620 raft_consensus.cc:3037] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.513078 9620 raft_consensus.cc:504] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:40.513396 9620 leader_election.cc:288] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 2 election: Requested vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:40.513684 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a1ce3e3c14323b261d9cbd4016a30" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:40.513794 9357 raft_consensus.cc:3032] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:58:40.513835 9357 raft_consensus.cc:729] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:40.513950 9357 consensus_queue.cc:260] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:40.514160 9357 raft_consensus.cc:3037] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.514189 9487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a1ce3e3c14323b261d9cbd4016a30" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:40.514292 9487 raft_consensus.cc:3037] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.514967 9487 raft_consensus.cc:2454] T b16a1ce3e3c14323b261d9cbd4016a30 P 717a0743a9b741f7909ac2557611bb26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 2.
I20240501 19:58:40.515208 9357 raft_consensus.cc:2454] T b16a1ce3e3c14323b261d9cbd4016a30 P 7b8ab321c9ce4518a42b35047ae72c15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 2.
I20240501 19:58:40.515550 9553 leader_election.cc:302] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:40.515707 10737 raft_consensus.cc:2781] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:58:40.515811 10737 raft_consensus.cc:686] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 LEADER]: Becoming Leader. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Running, Role: LEADER
I20240501 19:58:40.515923 10737 consensus_queue.cc:237] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } }
I20240501 19:58:40.516887 9619 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d04c802891964eaf8a73db2e807a7bf2"
dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
from {username='slave'} at 127.0.0.1:43300
I20240501 19:58:40.516983 9619 raft_consensus.cc:482] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:58:40.517033 9619 raft_consensus.cc:3037] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.517810 9619 raft_consensus.cc:504] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:40.518111 9619 leader_election.cc:288] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 2 election: Requested vote from peers 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2:37481), 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013)
I20240501 19:58:40.518353 9074 catalog_manager.cc:5484] T b16a1ce3e3c14323b261d9cbd4016a30 P a32eae53a7ae43a9a0295cc52fa16725 reported cstate change: term changed from 1 to 2, leader changed from 7b8ab321c9ce4518a42b35047ae72c15 (127.17.65.2) to a32eae53a7ae43a9a0295cc52fa16725 (127.17.65.4). New cstate: current_term: 2 leader_uuid: "a32eae53a7ae43a9a0295cc52fa16725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } }
I20240501 19:58:40.518491 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c802891964eaf8a73db2e807a7bf2" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "7b8ab321c9ce4518a42b35047ae72c15"
I20240501 19:58:40.518576 9357 raft_consensus.cc:3032] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:58:40.518611 9357 raft_consensus.cc:729] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b8ab321c9ce4518a42b35047ae72c15, State: Running, Role: LEADER
I20240501 19:58:40.518747 9357 consensus_queue.cc:260] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:40.518797 9487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04c802891964eaf8a73db2e807a7bf2" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:40.518889 9357 raft_consensus.cc:3037] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.518893 9487 raft_consensus.cc:3037] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.519631 9357 raft_consensus.cc:2454] T d04c802891964eaf8a73db2e807a7bf2 P 7b8ab321c9ce4518a42b35047ae72c15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 2.
I20240501 19:58:40.519811 9487 raft_consensus.cc:2454] T d04c802891964eaf8a73db2e807a7bf2 P 717a0743a9b741f7909ac2557611bb26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 2.
I20240501 19:58:40.519941 9553 leader_election.cc:302] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8ab321c9ce4518a42b35047ae72c15, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:40.520015 9618 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb15b158e91d42f5a993a53bffed32b6"
dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"
from {username='slave'} at 127.0.0.1:43300
I20240501 19:58:40.520071 11480 raft_consensus.cc:2781] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:58:40.520123 9618 raft_consensus.cc:482] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:58:40.520184 9618 raft_consensus.cc:3037] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.520206 11480 raft_consensus.cc:686] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 LEADER]: Becoming Leader. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Running, Role: LEADER
I20240501 19:58:40.520360 11480 consensus_queue.cc:237] T d04c802891964eaf8a73db2e807a7bf2 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8ab321c9ce4518a42b35047ae72c15" member_type: VOTER last_known_addr { host: "127.17.65.2" port: 37481 } } peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } }
I20240501 19:58:40.520903 9618 raft_consensus.cc:504] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:40.521196 9618 leader_election.cc:288] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 2 election: Requested vote from peers 717a0743a9b741f7909ac2557611bb26 (127.17.65.3:36013), 37fa3f77f5a644d181ccb7ea7ea15fff (127.17.65.1:36523)
I20240501 19:58:40.521519 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15b158e91d42f5a993a53bffed32b6" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff"
I20240501 19:58:40.521633 9226 raft_consensus.cc:3037] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.521888 9488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15b158e91d42f5a993a53bffed32b6" candidate_uuid: "a32eae53a7ae43a9a0295cc52fa16725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "717a0743a9b741f7909ac2557611bb26"
I20240501 19:58:40.521979 9488 raft_consensus.cc:3032] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:58:40.522022 9488 raft_consensus.cc:729] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 717a0743a9b741f7909ac2557611bb26, State: Running, Role: LEADER
I20240501 19:58:40.522156 9488 consensus_queue.cc:260] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:40.522408 9488 raft_consensus.cc:3037] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:58:40.522704 9226 raft_consensus.cc:2454] T fb15b158e91d42f5a993a53bffed32b6 P 37fa3f77f5a644d181ccb7ea7ea15fff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 2.
I20240501 19:58:40.522997 9554 leader_election.cc:302] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37fa3f77f5a644d181ccb7ea7ea15fff, a32eae53a7ae43a9a0295cc52fa16725; no voters:
I20240501 19:58:40.523196 11480 raft_consensus.cc:2781] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:58:40.523309 9488 raft_consensus.cc:2454] T fb15b158e91d42f5a993a53bffed32b6 P 717a0743a9b741f7909ac2557611bb26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a32eae53a7ae43a9a0295cc52fa16725 in term 2.
I20240501 19:58:40.523600 11480 raft_consensus.cc:686] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [term 2 LEADER]: Becoming Leader. State: Replica: a32eae53a7ae43a9a0295cc52fa16725, State: Running, Role: LEADER
I20240501 19:58:40.523774 11480 consensus_queue.cc:237] T fb15b158e91d42f5a993a53bffed32b6 P a32eae53a7ae43a9a0295cc52fa16725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717a0743a9b741f7909ac2557611bb26" member_type: VOTER last_known_addr { host: "127.17.65.3" port: 36013 } } peers { permanent_uuid: "a32eae53a7ae43a9a0295cc52fa16725" member_type: VOTER last_known_addr { host: "127.17.65.4" port: 45367 } } peers { permanent_uuid: "37fa3f77f5a644d181ccb7ea7ea15fff" member_type: VOTER last_known_addr { host: "127.17.65.1" port: 36523 } }
I20240501 19:58:40.524443 9618 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c28e44917cf44eab9118dda1f82991d0"
dest_uuid: "a32eae53a7ae43a9a0295cc52fa16725"